What happened
It was reported that in November 2024, a Stanford University professor specialising in misinformation submitted expert testimony in a US court case defending Minnesota’s law against election deepfakes, only for lawyers on the opposing side to discover that some of the academic studies he cited did not exist.
The professor had been engaged by the Minnesota Attorney General to explain how AI-generated misinformation could influence elections and undermine public trust, but while preparing his twelve-page declaration, he used GPT-4o to help compile references without independently checking every citation.
At least two studies were fabricated and another reference contained incorrect author information, prompting the opposing lawyers to challenge the reliability of the evidence. The professor acknowledged his mistake and said he had not intended to mislead the court, but in January 2025 the judge excluded his declaration and refused to allow a corrected version to be submitted.
What made the incident particularly striking was that an expert warning the court about the dangers of AI-generated misinformation had relied on information fabricated by AI, demonstrating that even specialist knowledge does not prevent mistakes when the information itself is not verified.
What an auditable version would have shown
An auditable system would have recorded which references were generated with AI assistance, whether the cited studies actually existed and what sources had been consulted to confirm their accuracy before the declaration was signed.
That record could have identified the fabricated citations before the document reached the court, rather than leaving the opposing lawyers to discover that the supporting research could not be found.
Where the gap was
Headlights identifies overconfidence and judgement surrendered, where someone places greater trust in information produced by a system than the available evidence justifies.
The professor understood the risks of AI misinformation, yet still accepted references that appeared academically credible without completing the checks normally expected when preparing expert evidence.
There is a familiar human behaviour behind this failure because expertise can create confidence in our ability to recognise mistakes, even when the technology produces information that looks convincing enough to escape closer scrutiny.
The problem was not the decision to use AI during research, but allowing its output to replace the independent verification required before making claims under oath.
What governance should have looked like
AI can assist with legal research and preparing documents, but anyone submitting evidence to a court remains responsible for ensuring that the authorities they cite are genuine and support the claims being made.
A CitationVerifier could check references against identifiable publications before they are included in a filing, while a ConductRecord could preserve evidence of the verification process and identify any citations that still require human review.
The broader lesson is that professional expertise does not remove the need to check AI-generated information, particularly when the consequences of an error extend beyond an individual document to the credibility of evidence presented in court.
Failure Pattern: An expert witness submitted a sworn declaration containing fabricated AI-generated academic citations, leading a federal judge to exclude the testimony.
Governance Principle: AI-assisted research used in legal proceedings must be independently verified against genuine sources, with clear responsibility for the accuracy of every citation submitted.
The reference implementation of CitationVerifier and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Order on Motion to Exclude Expert Testimony, Kohls v. Ellison, No. 0:24-cv-03754 (D. Minn., 10 January 2025) (Justia)
- Misinformation expert cites non-existent sources in Minnesota deep fake case (Minnesota Reformer)
- Stanford misinformation expert admits to ChatGPT ‘hallucinations’ in court statement (Stanford Daily)