What happened
It was reported that on 31 August 2020, Ronnie McNutt, a US Army veteran, died by suicide during a broadcast on Facebook Live. Reporting indicates Facebook took close to three hours to remove the original livestream despite user reports, and the company later said it was reviewing how it could have taken the stream down faster. Facebook also pointed to reduced moderation capacity during the COVID period, with fewer human reviewers available.
After the original came down, reporting indicates the footage did not stay contained, and clips spread to TikTok, YouTube, Instagram and Twitter. On TikTok the video reportedly surfaced on users’ For You pages through algorithmic recommendation, which put graphic content in front of people who had not gone looking for it. A known evasion technique made this harder to stop, because some clips began with harmless footage before cutting to the graphic moment, which slipped past automated detection tuned to the graphic frames.
It was reported that a friend and podcast co-host of McNutt’s, Josh Steen, started a campaign, #ReformForRonnie, that pressed the platforms to change how they handle this kind of material.
What an auditable version would have shown
An auditable version would show how long the live report queue actually took to reach a human during a period of reduced staffing, and what the automated systems scored on the edited reuploads that opened with innocuous footage. It would show whether the recommendation engine had any suppression rule for content under active review for self-harm, or whether engagement signals kept promoting it while moderation chased copies.
Where the gap was
There were two gaps. The first was removal speed on the original live broadcast, which was measured in hours when self-harm demands seconds. The second was that the same recommendation systems built to spread engaging content spread this too, and detection could be dodged by a simple edit to the opening seconds.
What governance should have looked like
A live stream of someone harming themselves cannot sit in an ordinary queue for close to three hours. It needs a fast lane to a real person, with a limit measured in minutes, and enough people on shift that a bad stretch for staffing does not turn minutes into hours. That is a verification gate. Two other things would have helped. The recommendation engine should stop pushing a video while it is under review for self-harm, instead of promoting it to strangers at the same moment moderators are trying to pull it down. And detection should recognise the clip itself, not just the graphic frames, so a harmless few seconds spliced onto the front does not walk it past the filter. A conduct record of how long removal took, and what was recommended while it was happening, would let anyone check the response instead of taking the company’s word for it.
The reference implementation of VerificationGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed, free for any company to install. The repository is public now.
Sources
- Graphic video of suicide spreads from Facebook to TikTok to YouTube as platforms fail moderation test (TechCrunch)
- How a suicide video on Facebook Live went viral on TikTok (Engadget)
Related
This entry concerns suicide and self-harm. If you or someone you know needs support in Australia, Lifeline is available on 13 11 14 and the Suicide Call Back Service on 1300 659 467.