What happened
On 9 January 2020, Detroit police drove to the suburb of Farmington Hills and arrested Robert Williams, a Black man, in his driveway as his wife and two young daughters watched. He was held for about thirty hours in a crowded cell. He had been identified as a suspect in the theft of watches from a Detroit Shinola store in 2018 by a facial-recognition system, made by the company DataWorks Plus, that the police had run a surveillance still through. He had not committed the theft, and his was the first case of a wrongful arrest driven by a false facial-recognition match to be reported publicly in the United States.
The chain that led to his driveway was thin at every link. A blurry image from the store’s security video was sent to be run against a state database of photographs; the system returned candidates, among them an expired driver’s-licence photo of Williams. That computer-generated suggestion was then placed in a photo lineup shown to a loss-prevention contractor who had not been present at the theft and had only seen the same grainy footage. The contractor picked Williams. On that basis a warrant issued and he was arrested. At no point before the arrest did anyone establish, by evidence independent of the match, that Williams had been in the store.
The case fell apart almost as soon as a person looked at it properly. In the interrogation room, shown the surveillance image, Williams held it next to his own face and said, “This is not me,” adding, “I hope you don’t think all Black people look alike.” The charge was dismissed. The American Civil Liberties Union, with the ACLU of Michigan and the University of Michigan Law School’s Civil Rights Litigation Initiative, sued the city in 2021. In June 2024 Detroit settled, agreeing to pay Williams 300,000 dollars and, more consequentially, to adopt some of the country’s tightest police limits on the technology: officers may not arrest on the strength of a facial-recognition match alone, may not build a witness lineup from a match without other evidence, must be trained on the technology’s tendency to misidentify people of colour, and must audit past cases in which face recognition had been used to obtain a warrant.
Williams was not the only one. Detroit’s use of the same technology produced at least two other wrongful arrests of Black residents, Michael Oliver in 2019 and Porcha Woodruff, eight months pregnant, in 2023, each following the same pattern of a match treated as an identification.
What an auditable version would have shown
The defect in Williams’s case was not that a face-matching algorithm produced a candidate; such systems are built to return the most similar faces, not to certify identity. The defect was that a candidate was carried forward as though it were a finding, with nothing in between checking it against the world. The match was a probabilistic suggestion from a low-quality image; the arrest treated it as probable cause.
An auditable version makes the difference between a lead and a fact explicit and recorded. It logs that an identification originated in a facial-recognition search, against what image, at what confidence, and it requires that before the suggestion drives a consequential action, an arrest, a lineup, it be verified against evidence independent of the match itself. With that, “the algorithm matched him” is recorded as the starting point of an investigation rather than its conclusion, and a case resting on a blurry image and a circular lineup is visibly unproven on the record before a person is taken from his home.
Where the gap was
The gap was that an unverified machine suggestion was allowed to stand in for probable cause, and that nothing recorded the suggestion’s true status as it moved through the case. A face-recognition hit is a hypothesis the system proposes; whether it is true is a question for a trusted source, not for the model that proposed it.
A VerificationGate is the control. A facial-recognition match does not authorise an action against a person until it has been checked against evidence the match did not itself generate, witnesses who saw the suspect, location data, anything that independently ties the person to the crime. A ConductRecord is the second: a record that an identification came from a facial-recognition search, with the image and confidence behind it, so that a court, a defendant and an auditor can see the basis on which a person was arrested, and so a match cannot quietly be laundered into a witness identification.
What governance should have looked like
The pattern recurs wherever a model’s output is treated as an answer rather than a proposal: the system is doing what it was built to do, returning a best guess, and the failure is the human process that forgets the guess is a guess. The match is a lead. Only verification makes it evidence.
The reference implementation of VerificationGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Williams v. City of Detroit (ACLU case page)
- Civil Rights Advocates Achieve the Nation’s Strongest Police Department Policy on Facial Recognition Technology (ACLU)
- Wrongfully Accused by an Algorithm (The New York Times)
- The new lawsuit that shows facial recognition is officially a civil rights issue (MIT Technology Review)