What happened
It was reported that Pa Edrissa Manjang, a courier who had worked for Uber Eats in Oxfordshire since November 2019, was repeatedly asked by the app to submit a live selfie so that his identity could be confirmed against the photographs already held on his account, and that in 2021 he lost access to the account after what the company described as continued mismatches. The check involved is Uber’s Real-Time ID Check, introduced in the United Kingdom in 2020 and built on Microsoft facial recognition technology, and both Microsoft and independent researchers have documented that facial recognition systems, including ones Microsoft has offered, carry higher error rates for some demographic groups, including some ethnic minorities. Manjang said the photographs he submitted were valid images of himself. In October 2021 he filed claims including indirect race discrimination, supported by the Equality and Human Rights Commission and the App Drivers and Couriers Union.
The case took nearly three years. According to the EHRC, Uber’s application to have the claim struck out was refused in May 2022, a reconsideration hearing in September 2023 upheld that decision, and a seventeen-day final hearing was listed for November 2024 before the parties settled in March 2024 on terms that were not disclosed, which means there is no tribunal finding that discrimination occurred. Baroness Kishwer Falkner of the EHRC said that AI is complex and presents unique challenges for employers, lawyers and regulators, and that the Commission was particularly concerned that Mr Manjang was not made aware his account was in the process of deactivation, nor provided any clear and effective route to challenge the technology. Uber was reported as saying that its Real-Time ID Check is designed to help keep everyone who uses the app safe and includes robust human review so that decisions about someone’s livelihood are not made in a vacuum without oversight, and that automated facial verification was not, in its view, the reason for Mr Manjang’s temporary loss of access. He was reinstated and has continued to deliver for the platform.
What an auditable version would have shown
Uber says its check includes robust human review. From the outside, robust human review and a click on a screen look the same. What would tell them apart is a record of the decision: who looked at the rejected photographs, what they saw, and why they concluded the account should end. The other record Uber has not published is a standing measurement of how the check performs across demographic groups, so a pattern of failures shows up as a number the company already holds, rather than as an allegation a courier has to fund a tribunal claim to test.
Where the gap was
Manjang says he sent valid photographs of himself and the app kept rejecting them. He could not see how the check worked, could not correct it, and had no way to challenge it before he lost the account. A MetricRecord counts how often the check fails, and for whom, so a pattern shows up inside the company as a number rather than in a tribunal three years later. A ConductRecord keeps an account of what the system did and what any human reviewer did about it, in enough detail to show the review was real. Without that, an appeal route exists on paper and nowhere else. Identity verification has legitimate safety purposes. What the Commission objected to was that Manjang was, in its words, neither made aware his account was being deactivated nor given any clear and effective route to challenge the technology.
What governance should have looked like
Manjang lost access to his account, and the income that came with it, on the strength of a check he was never shown. Where a check can do that, three things should follow. Its performance should be measured by group and kept as a standing figure, not an average that hides the people it fails most often. The person should be told when their account is at risk. And the decision to cut them off should be a documented human judgement they can see and contest. It took an equality regulator, a union and nearly three years of litigation to reach a settlement, and even then no explanation of why the app kept rejecting his photographs was ever put on the record.
Failure Pattern: a worker’s access to his livelihood was governed by a biometric check documented to fail more often for some ethnic minorities, with no standing measurement of how it performed by group and no meaningful, recorded human decision before he was cut off.
Governance Principle: where an automated check can end someone’s ability to earn, its error rates should be measured and published by group, and the decision to act on it should be a real human decision that is recorded and can be challenged.
The reference implementation of MetricRecord and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Uber Eats courier wins payout with help of equality watchdog, after facing problematic AI checks (Equality and Human Rights Commission)
- Payout for Uber Eats driver over face scan bias case (BBC News)
- Uber Eats courier’s fight against AI bias shows justice under UK law is hard won (TechCrunch)