What happened
It was reported that in July 2018 the American Civil Liberties Union tested Amazon’s face-matching tool, Rekognition. The ACLU reported that it ran photos of all members of the US Congress against 25,000 publicly available arrest photos, using the tool’s default settings and an 80 percent confidence threshold, and that the tool falsely matched 28 members of Congress to police mugshots. The ACLU said nearly 40 percent of those false matches were people of colour, even though they made up about 20 percent of Congress, and that six of those wrongly matched were members of the Congressional Black Caucus. The whole test, it noted, cost 12.33 dollars.
Amazon said the ACLU had used the default 80 percent setting, and that for law enforcement it recommended a higher confidence threshold and human review, while critics pointed out that the default was the setting many customers would use out of the box. In June 2020, amid wider protests over policing, Amazon announced a one-year moratorium on police use of Rekognition, and in 2021 it said the moratorium would continue indefinitely. It continued offering Rekognition for other, non-police image-analysis uses.
What an auditable version would have shown
If you are going to use a tool to identify people, you need to know how often it gets that identification wrong. That means testing the settings customers actually use and breaking the results down by skin tone, age and sex, so a buyer can see the false-match rate rather than just a confidence score attached to one result, and so a match is presented as something to investigate rather than as a fact. Those records would have shown buyers what happened at the default setting, including who was more likely to be falsely matched, before the technology was ever pointed at real people.
Where the gap was
The problem was not simply that Rekognition made mistakes. It was what the person using it could see when it did. A confident-looking match appeared without a clear explanation of how often the system could be wrong, the default setting produced false matches, and nothing in front of the buyer showed that those errors fell more heavily on people of colour. When the consequence of a wrong identification could involve the police, that information matters.
What governance should have looked like
Before a face-matching tool is used to identify people, it should be tested for accuracy and bias using the same settings customers will actually run. A VerificationGate is designed to hold a tool like this against accuracy and fairness tests before it reaches a setting where a wrong match matters, and if it does not pass, it does not move forward. A MetricRecord would keep the evidence behind that decision, including match and error rates broken down by group, so buyers and regulators could see the numbers rather than relying on a general claim of accuracy. The question should be answered before deployment: how often is this system wrong, and who is it getting wrong?
The reference implementation of VerificationGate and MetricRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed, free for any company to install. The repository is public now.
Sources
- Amazon’s Face Recognition Falsely Matched 28 Members of Congress With Mugshots (ACLU)
- Amazon facial AI matched politicians with criminals in ACLU test (Bloomberg)
- ACLU says Amazon’s facial recognition matched Congress members to mugshots (TechCrunch)