What happened
COMPAS, short for Correctional Offender Management Profiling for Alternative Sanctions, is a risk-assessment tool sold by the company Northpointe, later renamed equivant, and used across the United States to help courts decide on bail, sentencing and parole. It scores a defendant’s likelihood of reoffending from the answers to a 137-question survey. Race is not one of the questions.
In May 2016 the newsroom ProPublica published an analysis of COMPAS scores for more than seven thousand people arrested in Broward County, Florida, checked against who actually reoffended over the following two years. Measured one way, the tool was roughly as accurate for Black and white defendants, predicting reoffending correctly about six times in ten for each group. Its errors, though, fell in opposite directions by race. Black defendants who did not go on to reoffend had been labelled high risk at nearly twice the rate of white defendants in the same position, about 45 per cent against 23 per cent. White defendants who did reoffend had been labelled low risk far more often than comparable Black defendants, about 48 per cent against 28 per cent. The people the tool was wrong about were disproportionately Black in the direction that does harm, and disproportionately white in the direction that does not.
Northpointe rejected the analysis, arguing that within any given score its predictions held roughly equally well regardless of race, a different and also defensible definition of fairness. The dispute turned out to rest on a mathematical fact later set out formally by several researchers: when two groups reoffend at different underlying rates, no score can be equally calibrated and produce equal error rates at the same time. Both ProPublica and Northpointe were measuring something real. Neither measure was being reported to the courts using the scores.
The tool’s workings were a trade secret. In State v. Loomis, decided by the Wisconsin Supreme Court in July 2016, a man sentenced with the help of a COMPAS score argued that he could not challenge a number he was not allowed to inspect. The court allowed COMPAS to keep being used in sentencing, with cautions that a score must not be the determinative factor, and the United States Supreme Court declined to hear the case in 2017. The tool stayed proprietary and in use.
What an auditable version would have shown
A court using a risk score needs to answer two questions that COMPAS could not. The first is how the score for this defendant was reached, which the trade secret put out of reach. The second is whether the tool’s errors fall unevenly across groups, and by how much, which nobody was measuring as the scores were handed up. An auditable version records each assessment with the inputs behind it, and computes a standing, recomputable measure of error rates across groups that a court or an oversight body can verify for itself. With that, the disparity ProPublica spent months extracting from public records is a number visible from inside the system, and the unavoidable trade-off between calibration and equal error rates is something a court chooses with its eyes open rather than a property nobody has looked at.
Where the gap was
The harm here was not a single wrong score but a system that produced unequal errors at scale while disclosing neither how it worked nor how it failed. A MetricRecord is the control on the second: a signed measure of false-positive and false-negative rates across protected groups, recomputable from the underlying assessments, so disparate impact is monitored rather than discovered by journalists. A ConductRecord is the control on the first: a record of each assessment, its inputs and its score, so a defendant facing a number can see and contest its basis instead of being told it is confidential. Neither control resolves the mathematical tension between competing fairness definitions. What they do is make the tension, and the choice it forces, visible to the people whose liberty depends on it.
What governance should have looked like
A score that influences whether a person is jailed is exercising a portion of the state’s most serious power, and it was held to a lower standard of scrutiny than the human decisions around it. The lesson is that opacity and consequence cannot sit together: a tool used in sentencing must expose how it reached a given score, must be measured continuously for unequal error across groups, and must leave a record an individual can challenge. The hard part is not the measurement. It is accepting that some fairness properties cannot all be satisfied at once, and that the choice between them belongs in the open, recorded and reviewable, rather than inside a vendor’s trade secret.
The reference implementation of MetricRecord and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Machine Bias (ProPublica)
- How We Analyzed the COMPAS Recidivism Algorithm (ProPublica)
- State v. Loomis, 881 N.W.2d 749 (Wis. 2016) (Justia)
- Bias in Criminal Risk Scores Is Mathematically Inevitable, Researchers Say (ProPublica)