What happened
The case, Kohls v. Ellison, was a challenge to Minnesota’s 2023 law restricting the use of deepfakes to influence an election, brought by a conservative satirist and a state representative. The Minnesota Attorney General was defending the law, and engaged Jeff Hancock, a Stanford communication professor and a well-known scholar of misinformation, as an expert. On 1 November 2024 Hancock filed a twelve-page declaration arguing that deepfakes amplify misinformation and erode trust in democratic institutions.
The declaration cited a number of sources. At least two of them could not be found, because they did not exist, and a third misattributed its authors. Hancock had used GPT-4o, alongside Google Scholar, to help assemble the citation list, and had not verified the entries before signing the declaration under oath. The opposing lawyers could not locate the studies and brought it to the court’s attention. Hancock acknowledged that the citations were AI-generated and unchecked, and wrote that he had not intended to mislead the court or counsel. In an order dated 10 January 2025, Judge Laura Provinzino excluded the declaration, writing that his citation to fake, AI-generated sources “shatters his credibility with this Court.” She also denied the Attorney General leave to file an amended version, so the exclusion was final rather than a setback the state could repair.
The irony was noted widely, and it is the point. A recognised authority on AI-driven misinformation had filed, in defence of a law against AI-driven deception, citations that an AI had fabricated. The judge made the point herself, writing that Hancock had “fallen victim to the siren call of relying too heavily on AI” in a case about the dangers of AI. Expertise in the subject did not protect him, because the missing step was not knowledge. It was verification.
What an auditable version would have shown
The professional duty in a sworn filing is old and unambiguous: the authorities you cite must be real and must say what you claim. What was missing was any record connecting each citation to a confirmed source. An auditable version logs every citation as it is assembled, with a status of verified, partially verified, or unverified and the source consulted, and checks the final document against that log before it is filed. A citation still marked unverified blocks the filing. With that record, the two fabricated entries would have surfaced at the desk, before the declaration was signed, rather than in the opposing brief.
Where the gap was
The gap is the same one running through the legal cases in this library: a generative tool returns plausible, correctly formatted citations whether or not the underlying work exists, and the lookup that used to guarantee a real source has been removed without anyone noticing. The control is a CitationVerifier: every citation produced with AI assistance is checked against a primary source before the document leaves the desk, and an unverifiable citation stops the document. A ConductRecord preserves the prompts, the model version, the citations, and the verification result, so a practitioner can show how each authority was confirmed.
What governance should have looked like
The defence is not that experts should avoid AI. It is a verification gate built into the work, that no filing passes without, plus the duty to disclose AI use and to be able to produce the record of how citations were checked. That record is what restores the assumption a court has always relied on, that a cited authority has been confirmed against a real source by someone. The signed record is what makes the disclosure verifiable rather than a promise.
The reference implementation of CitationVerifier and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.