What happened
In December 2023, researchers at the Stanford Internet Observatory examined LAION-5B, a public dataset containing more than five billion image-text pairs that had been used to train AI image generators, including Stable Diffusion.
What they found was deeply concerning.
The researchers identified 3,226 suspected instances of child sexual abuse material, with about 1,008 later validated by external child safety organisations. They used automated detection methods, including matching against databases of known illegal material, and worked with specialist organisations to confirm and report what they found. The process was designed to minimise direct exposure to the material.
LAION said it had a strict policy against this content and took the dataset offline while it investigated. It later released a cleaned version called Re-LAION-5B.
The incident highlighted a much bigger problem. The dataset was so large that nobody had fully checked what was inside before it became training data for AI models. Researchers warned that if prohibited material enters a training dataset, it may influence the behaviour of models trained on it, even if the impact on any individual model cannot be measured afterwards.
What an auditable version would have shown
An auditable training dataset should tell the story of where each piece of data came from. It should show what checks were carried out before the data was added, what screening tools were used, what was flagged, what was removed, and who approved the dataset for training. That way, if someone later asks what a model was trained on, the answer already exists. It doesn’t have to be discovered by researchers months or years later.
Where the gap was
The problem wasn’t simply that prohibited material existed. It was that the dataset had grown to a size where nobody could confidently say what was inside it. Collecting billions of images from the open internet is relatively easy. Proving each one of those images is suitable for training is much harder. The scale that made the dataset valuable also made it difficult to verify.
What governance should have looked like
Training data should be treated like any other critical asset. Organisations should know where it came from, what checks were performed before it was included, and who approved it for use. Each new item should be screened against known databases of prohibited material before it becomes part of a training dataset, with a clear record of what was blocked, what was reviewed, and why. The goal isn’t to slow innovation. It’s to make sure organisations can answer a simple question with evidence: what exactly did this model learn from?
A ConductRecord of where each part of a dataset came from, and a ConstraintGate that screens incoming material against known-bad sources and blocks what it matches, would mean prohibited content is caught at the door rather than found later inside a shipped model.
The reference implementation of ConductRecord and ConstraintGate is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed, free for any company to install. The repository is public now.
Sources
- Hundreds of images of child sexual abuse material were found in a massive dataset used to train AI image-generating tools (CNN)
- Largest dataset powering AI images removed after the discovery of child sexual abuse material (404 Media)
- LAION releases the AI dataset Re-LAION-5B purged of links to child abuse images (The Decoder)