What happened
In November 2022, Meta, together with Papers with Code, released a public demo of Galactica, a large language model trained on around 48 million scientific papers, textbooks, websites, lecture notes and reference works. It was designed to help people summarise research, solve maths problems, write scientific code and explore scientific ideas.
The demo didn’t last long.
Within hours, researchers began sharing examples of Galactica confidently generating information that looked like genuine scientific writing but wasn’t true. It invented studies, fabricated citations that appeared real, sometimes attributed them to real researchers, and produced false or biased answers on some topics. Because the responses looked like academic papers, many of the mistakes weren’t obvious at first glance.
Just three days after launch, Meta took the public demo offline, saying it could no longer support it. The underlying model, however, remained available for research.
What an auditable version would have shown
An auditable system wouldn’t just generate citations. It would show whether those citations had actually been checked. For each reference, there would be a record confirming that the paper exists, that the authors are real, and that the source actually supports the claim being made. If a citation couldn’t be verified, the model would either say so or avoid presenting it as fact. That way, readers could tell the difference between evidence and something the model had simply invented.
Where the gap was
The problem wasn’t that the model wrote fluently. The problem was that it wrote convincingly. People naturally associate academic language and references with credibility. Galactica could produce text that looked like science even when the evidence behind it didn’t exist. There was nothing stopping a fabricated citation from being presented as though it were genuine.
What governance should have looked like
If an AI system generates facts and citations, it should verify them before presenting them as evidence. Each citation should be checked to confirm the source exists and supports the claim attached to it. If it can’t be verified, the system should clearly say so instead of presenting it with confidence. Before a public release, the model should also be tested using prompts designed to expose fabricated references, misleading claims and other high-risk failures. Those results should become part of the launch decision, not something discovered by users after release. The goal isn’t to stop AI from helping people research. It’s to make sure that when an AI presents something as evidence, people can trust that it has been checked.
A CitationVerifier is designed to confirm that a source exists and supports the claim attached to it, and to hold back or mark anything it cannot stand behind. A VerificationGate before a public release is designed to test how a tool behaves on the questions where a confident wrong answer does the most harm, and to hold the launch until it passes.
The reference implementation of CitationVerifier and VerificationGate is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed, free for any company to install. The repository is public now.
Sources
- Why Meta’s latest large language model only survived three days online (MIT Technology Review)
- Meta’s ‘biased’ science-writing AI demo gets pulled after three days (Silicon Republic)