What happened
It was reported that São Paulo built a mass surveillance system called Smart Sampa, a network planned to grow to around twenty thousand cameras, many of them using facial recognition to match people on the street against police and court databases in real time. The contract was awarded in 2023 and the system was rolled out from 2024. The city government and the police said the cameras helped them find people who were wanted, and pointed to arrests the system had made.
It was reported that the same system also stopped people who had done nothing wrong. Journalists documented cases where the software matched an innocent person to someone on a watchlist, and the person was approached by police on the strength of that match. Civil society groups and researchers warned that facial recognition misidentifies darker-skinned faces more often than lighter ones, and noted that in Brazil the great majority of people arrested through facial recognition have been Black, so the burden of a wrong match would not fall evenly. A person the system matched had no way to know they had been flagged, or to see what the match rested on, before being stopped.
What an auditable version would have shown
A facial recognition match is the software’s claim that the person in front of the camera is someone on a watchlist, and that claim can be wrong. The question an audit answers is how often it is wrong, and what happens when it is. An auditable version keeps a record of every match it acts on: the image, the watchlist entry it was matched to, and whether a person checked it before anyone was stopped, so a wrong match can be traced and put right. It would also count how often matches turn out wrong, and whether they go wrong more often for some groups than others, so the system’s accuracy, and any racial skew in its mistakes, is something the city can measure rather than something its residents can only fear.
Where the gap was
People were stopped on the strength of a facial recognition match, and a person wrongly matched could not know it had happened or see what it rested on. A ConstraintGate treats a match as a lead to be checked, not a fact to act on, so a person is not stopped until the match has been confirmed against a reliable source. A ConductRecord keeps each match, the image and the watchlist entry, so a wrong match can be found and undone. A MetricRecord counts how often matches are wrong, and whether the errors fall harder on some groups, so the system’s accuracy and any racial skew are measured rather than assumed. The safeguard cannot be the confidence of the match. It has to be the check that comes before anyone is stopped.
What governance should have looked like
Where facial recognition is used to act on people in public, a match should be checked before anyone is stopped, not acted on as if it were already certain, and the person should not be approached until it has been confirmed against a reliable source. Best practice would be for the operator to record every match it acts on and the basis for it, and to measure and publish how often matches are wrong and whether the errors fall more heavily on some groups than others. São Paulo pointed to the arrests the system had helped make. Whether it also stopped innocent people, and whether it went wrong more often for some groups than others, is not answered by a count of arrests but by a record of the matches behind them.
Failure Pattern: a city matched people on the street against watchlists by facial recognition and acted on the matches, and a person wrongly matched could not know they had been flagged or see the basis before being stopped.
Governance Principle: where facial recognition is used to act on people in public, a match must be verified against a reliable source before anyone is stopped, each match and its basis must be recorded, and the error rate across groups must be measured and published.
The reference implementation of ConstraintGate, ConductRecord and MetricRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- São Paulo AI policing nabs criminals, and a few innocents (Digital Journal / AFP)
- Facial recognition surveillance in São Paulo could worsen racism (Al Jazeera)
- 1.9M a month bid wins contract for São Paulo 20K camera biometric surveillance system (Biometric Update)