HD-INC-131

Social media and online platforms · New Zealand · 2019 · Automated detection failure on live video

A terror attack was livestreamed on Facebook for 17 minutes and the platform's automated detection did not catch it while it ran

By Ellie Harris · Filed Attack livestreamed 15 March 2019

Alleged: Facebook (now Meta) developed or deployed the AI system implicated in this incident. Details are drawn from public reports; parties are presumed innocent of any wrongdoing not established by an official finding.

What happened

It was reported that on 15 March 2019, a gunman attacked two mosques in Christchurch, New Zealand, and livestreamed the first attack on Facebook. The broadcast ran for around 17 minutes.

Facebook reportedly said the live video was watched by fewer than 200 people while it was happening, and by around 4,000 people in total before it was removed. No user reported the stream during the live broadcast, and the first user report came about 12 minutes after the stream had already ended. Facebook said it removed roughly 1.5 million copies of the video worldwide in the first 24 hours, with about 1.2 million of those blocked at the point of upload.

Facebook also said its automated systems did not flag the video while it ran. The platform leaned on the people watching to report it, and no one did while it was live. What spread afterwards was not one video but a flood of reuploaded copies that the same systems then had to chase.

Facebook later said it worked with law enforcement, including on firearms training footage, to help its systems better recognise this kind of first person attack video in future.

What an auditable version would have shown

An auditable version of this system would show, in a record you could inspect afterwards, what the automated classifiers actually saw during those 17 minutes, what confidence they returned, and why the footage did not cross a threshold for human review. It would show whether live video was being scanned with the same rigour as uploaded video, and whether any escalation path existed that did not wait on a member of the public to press report. Without that record, the failure can only be argued about from press statements rather than traced.

Where the gap was

The gap was in detection, not in review. Reporting at the time indicated the systems had little training data for that kind of first person, weapon forward footage, so the classifiers did not flag it as the atrocity it was. The design leaned heavily on users to report live content, and the people watching this stream were not the people who would report it.

What governance should have looked like

Live video is not the same problem as an uploaded clip, and it should not have been left to the same tools. When a stream sets off a serious warning sign, it needs to reach a person while it is still running, not after someone in the audience decides to report it. That is what a verification gate is for. And there should have been a plain record of what the detection systems saw during those 17 minutes, so afterwards the story could be told from the logs instead of a press statement. That is what a conduct record keeps.

The reference implementation of VerificationGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed, free for any company to install. The repository is public now.

Sources

The mailing list

Fresh incident reports every week. One email to match.

We add new incidents to the library regularly, and send a single short email each week with what's new. The library stays free and open; this is just how you keep up with it.

No tracking. Unsubscribe in one click.

The record

An auditable system would have produced a signed, tamper-evident record the moment this happened: what the system did, the version that did it, the basis it acted on, and the action taken, and Facebook (now Meta) could have produced it on demand.

This is the record the system as deployed did not produce in a signed, auditable form.

What this teaches

Capture what happened when it happens

What the system did, the version that did it, the basis it acted on, and the action taken, recorded at the moment, not reconstructed after.

Sign it, so no one has to trust the record-keeper

A tamper-evident entry. Edit it later and the signature breaks. The record does not ask for the benefit of the doubt.

Make it verifiable by anyone

A court, a regulator, a customer's lawyer can check the record themselves, without taking the company, or us, at our word.

Also in the library

HD-INC-001 Air Canada chatbot promised a bereavement refund policy that did not exist Aviation · 2022 HD-INC-002 Mata v. Avianca, the lawyer who cited six cases that did not exist and asked ChatGPT to confirm them Legal services · 2023 HD-INC-003 Michael Cohen gave his lawyer fake case citations he had got from Google Bard, and his lawyer filed them in a federal court Legal services · 2023

Headlights summarises publicly reported AI incidents. All summaries are independently written, attributed to their original sources, and intended for research and educational purposes. Allegations are identified as such until established through official findings.

This report is based on the sources listed above and reflects information available at the time of review; later developments may not be captured. Where a person is described as charged with or alleged to have done something, that allegation is unproven unless a conviction or a court or regulatory finding is stated. Headlights publishes journalism and commentary, not legal advice.

Want to write back?

Direct to my inbox.

ellie@useheadlights.com →