What happened
Between 2012 and 2020, the pharmacy chain Rite Aid ran facial recognition technology in hundreds of its stores, scanning the faces of people as they shopped and matching them against a watchlist of individuals it considered likely to shoplift or cause trouble. It did not tell customers this was happening, and according to the Federal Trade Commission it instructed employees not to reveal it.
The watchlist was built from tens of thousands of images, many of them low-quality stills from security cameras, staff phones, and even news reports, and Rite Aid never seriously tested whether the matching worked. It did not, often. The system generated thousands of false-positive matches. When it flagged someone, employees followed them through the shop, searched them, ordered them out, called the police, and accused them, sometimes in front of family, of crimes they had not committed. In one case described by the FTC, an eleven-year-old girl was stopped and searched because of a false match, and her mother said she missed work because the child was so distressed. In another, staff called the police on a Black customer after the system matched her to an alert image later described as “a white lady with blonde hair.”
The harm was not evenly spread. The FTC found that the technology produced more false matches in stores in plurality-Black and plurality-Asian communities, and that Rite Aid was more likely to install it in non-white areas in the first place. Black, Asian and Latino customers, and women, were especially likely to be hurt by it.
On 19 December 2023 the FTC announced a settlement. Rite Aid is barred from using facial recognition for surveillance for five years, must delete the images it collected and any models built from them, must tell customers when the technology is in use and when it acts against them, and must run a monitoring program overseen by outside assessors. Samuel Levine, head of the FTC’s Bureau of Consumer Protection, said Rite Aid’s “reckless use of facial surveillance systems left its customers facing humiliation and other harms.” Because the company was by then in bankruptcy, the order was subject to the bankruptcy court’s approval.
What an auditable version would have shown
Every match the system made and every action a store took on the back of it could have been recorded, counted and examined. The failure was that none of it was: Rite Aid did not measure how often the technology was wrong, did not track who its errors fell on, and kept no account a wrongly accused customer could point to. An auditable version records each match, the confidence behind it and the action taken, and computes a standing measure of false-match rates across groups. With that, a system generating thousands of false positives, concentrated on particular communities, is a monitored number that should have stopped the deployment, rather than a pattern the regulator had to assemble years later from the wreckage.
Where the gap was
Three controls were missing. Nothing stopped staff acting on a weak or untested match, nothing measured how the errors were distributed, and nothing recorded the matches in a way an accused customer could contest. A ConstraintGate is the control on the first: an action against a person does not proceed on a match that has not cleared a confidence threshold and an accuracy standard the system is held to. A MetricRecord is the control on the second: a signed, recomputable measure of false-match rates across groups, so disparate harm is visible as monitoring rather than as a later finding. A ConductRecord is the control on the third: a record of each match and what was done about it, so a person searched on the strength of a bad match has something to point to instead of a denial. A watchlist that scans everyone and is wrong thousands of times is exactly the kind of high-consequence automation that should not act without a verified basis and a record.
What governance should have looked like
Pointing facial recognition at every shopper turns an ordinary errand into a search, and doing it without notice, without testing, and without measurement turns the system’s errors into accusations against innocent people. The lesson is that consequence sets the standard: a technology that can have a customer followed and searched needed a confidence threshold below which no one is acted upon, continuous measurement of who its mistakes land on, a recorded and contestable basis for each match, and customers told it was there at all. Rite Aid had none of these, and the people who paid for it were disproportionately those already most likely to be treated as suspects.
The reference implementation of ConstraintGate, MetricRecord, and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Rite Aid Banned from Using AI Facial Recognition After FTC Says Retailer Deployed Technology without Reasonable Safeguards (Federal Trade Commission)
- Coming face to face with Rite Aid’s allegedly unfair use of facial recognition technology (FTC Business Blog)
- Statement of Commissioner Alvaro M. Bedoya on FTC v. Rite Aid (FTC, PDF)