# HD-INC-140: Meta released an AI built to write science, and pulled the demo after three days when it produced confident, made-up results

> In November 2022 Meta released a public demo of Galactica, a large language model trained on scientific papers and meant to help write and summarise research. Within days, researchers showed it would produce authoritative-sounding text that was wrong, including invented studies and citations, and biased content presented as fact. Meta took the demo down after about three days. The model sounded most convincing when it was making things up.

- Source: https://useheadlights.com/library/HD-INC-140/
- Publisher: Headlights (useheadlights.com), author Ellie Harris
- Published: 2026-08-11
- Company: Meta Platforms, Inc. (Meta AI)
- Sector: Consumer AI
- Country: United States
- Year: 2022
- Behaviour in the record (Headlights reading): Overconfidence
- Governance domains: Consumer protection
- Licence: CC BY-NC 4.0. Cite as: Harris, E. (2026). Meta released an AI built to write science, and pulled the demo after three days when it produced confident, made-up results. Headlights AI Incident Library, HD-INC-140. https://useheadlights.com/library/HD-INC-140/

## What happened

In November 2022, Meta, together with Papers with Code, released a public demo of Galactica, a large language model trained on around 48 million scientific papers, textbooks, websites, lecture notes and reference works. It was designed to help people summarise research, solve maths problems, write scientific code and explore scientific ideas.

The demo didn't last long.

Within hours, researchers began sharing examples of Galactica confidently generating information that looked like genuine scientific writing but wasn't true. It invented studies, fabricated citations that appeared real, sometimes attributed them to real researchers, and produced false or biased answers on some topics. Because the responses looked like academic papers, many of the mistakes weren't obvious at first glance.

Just three days after launch, Meta took the public demo offline, saying it could no longer support it. The underlying model, however, remained available for research.

## What an auditable version would have shown

An auditable system wouldn't just generate citations. It would show whether those citations had actually been checked. For each reference, there would be a record confirming that the paper exists, that the authors are real, and that the source actually supports the claim being made. If a citation couldn't be verified, the model would either say so or avoid presenting it as fact. That way, readers could tell the difference between evidence and something the model had simply invented.

## Where the gap was

The problem wasn't that the model wrote fluently. The problem was that it wrote convincingly. People naturally associate academic language and references with credibility. Galactica could produce text that looked like science even when the evidence behind it didn't exist. There was nothing stopping a fabricated citation from being presented as though it were genuine.

## What governance should have looked like

If an AI system generates facts and citations, it should verify them before presenting them as evidence. Each citation should be checked to confirm the source exists and supports the claim attached to it. If it can't be verified, the system should clearly say so instead of presenting it with confidence. Before a public release, the model should also be tested using prompts designed to expose fabricated references, misleading claims and other high-risk failures. Those results should become part of the launch decision, not something discovered by users after release. The goal isn't to stop AI from helping people research. It's to make sure that when an AI presents something as evidence, people can trust that it has been checked.

A CitationVerifier is designed to confirm that a source exists and supports the claim attached to it, and to hold back or mark anything it cannot stand behind. A VerificationGate before a public release is designed to test how a tool behaves on the questions where a confident wrong answer does the most harm, and to hold the launch until it passes.

The reference implementation of CitationVerifier and VerificationGate is open source. It lives at [github.com/saffronandindia/headlights-oss](https://github.com/saffronandindia/headlights-oss), Apache 2.0 licensed, free for any company to install. The repository is public now.

## Sources

- [Why Meta's latest large language model only survived three days online (MIT Technology Review)](https://www.technologyreview.com/2022/11/18/1063487/meta-large-language-model-ai-only-survived-three-days-gpt-3-science/)
- [Meta's 'biased' science-writing AI demo gets pulled after three days (Silicon Republic)](https://www.siliconrepublic.com/machines/galactica-meta-ai-large-language-model)

## Related

- [HD-INC-034](/library/HD-INC-034)
- [HD-INC-029](/library/HD-INC-029)
- [HD-INC-049](/library/HD-INC-049)