What happened
It was reported in early September 2021 that Facebook users who watched a video posted by the Daily Mail in June 2020, showing an encounter between a white man and a group of Black men, were shown an automatically generated prompt from the platform’s recommendation system asking whether they would like to keep seeing videos about Primates. The video was a news clip and had nothing to do with animals or primates of any kind. It became public when Darci Groves, a former Facebook content design manager, was sent a screenshot by a friend and posted it, describing the prompt as unacceptable and egregious, and the New York Times reported it the same week.
A Facebook spokesperson, quoted by NPR and Al Jazeera, said the company apologised to anyone who may have seen the offensive recommendations, called it a clearly unacceptable error, and said Facebook had disabled the entire topic recommendation feature as soon as it realised what was happening so that it could investigate the cause and prevent it from happening again. The failure did not come to light through monitoring, an audit, or a user report escalated inside the company. It came to light through a screenshot that reached a former employee.
What an auditable version would have shown
Facebook has not said how often the prompt appeared, and nothing published suggests anyone was counting. A classifier that labels people should be tested on people before its labels are used, and the result kept as a running figure rather than a launch-day check nobody looks at again. The labels themselves should be sampled, so a category like this one turns up in Facebook’s own review rather than in a screenshot sent to a former employee.
Where the gap was
A vision model applied a dehumanising label to a video of Black men and the platform passed it straight through to users as a suggestion. On the public record, nothing inside the company caught it before a screenshot did. A MetricRecord counts how often the classifier is wrong, and about whom, as a matter of routine, so a disparity is a number the company already holds. A VerificationGate holds a capability claim to evidence before anyone relies on it. A model’s accuracy at labelling images of people has to be demonstrated before its labels go in front of a global audience. Classifiers make errors. On the public record, nothing inside Facebook measured this class of error on people, and the label reached the public with nothing standing in between.
What governance should have looked like
A system that labels images of people should have its accuracy on people measured across groups before those labels are used, continuously rather than once at launch, and categories capable of dehumanising anyone should be a hard limit on what may be shown rather than a probability to be weighed. Facebook found out the way everybody else did, from a screenshot. A company that cannot detect this in its own systems cannot say how often it happens.
Failure Pattern: a vision system applied a dehumanising label to people and the platform surfaced it to users, with nothing measuring how the classifier performed across groups and no check standing between its labels and the public.
Governance Principle: before a classifier’s labels are shown to users, its accuracy should be measured across the groups it will be applied to and held to that evidence, and the results should be a standing figure rather than something discovered when a screenshot circulates.
The reference implementation of MetricRecord and VerificationGate is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Facebook apologizes after its AI labels Black men as ‘primates’ (NPR)
- Facebook apology as AI labels black men ‘primates’ (BBC News)
- Facebook apologises for labelling video of Black men ‘primates’ (Al Jazeera)