What happened
It was reported that on 29 November 2023, a lawyer filed a motion in the United States District Court for the Southern District of New York asking for his client’s supervised release to end early. The client had completed a federal prison sentence following campaign-finance and tax-related convictions connected to the 2016 presidential election and wanted to persuade the court that there were legal grounds to shorten the remaining period of supervision.
To support the request, the lawyer cited three decisions that appeared to show other federal courts had granted early termination in similar circumstances. The cases were United States v. Figueroa-Florez, United States v. Ortiz and United States v. Amato, each accompanied by what looked like a genuine legal citation. There was a problem with all three: none corresponded to the decisions described in the motion. One citation pointed to an unrelated Fourth Circuit decision, another referred to a veterans’ appeals decision, and the third could not be found.
The citations had come from the client, who had used Google’s Bard chatbot to look for legal precedents. He later explained that he believed Bard worked like a “super-charged search engine” and assumed it was retrieving genuine court decisions. He passed the results to his lawyer without explaining that they had been generated by an AI chatbot, and the lawyer included them in the motion without independently checking whether the cases existed.
On 12 December 2023, the judge ordered the lawyer to produce copies of the three decisions, but he was unable to do so. The court subsequently required an explanation, and on 29 December the client submitted a sworn affidavit acknowledging that he had used Bard and had not understood that the system could invent legal citations. The lawyer also submitted an affidavit explaining that he believed the citations had come from another lawyer who had commented on an earlier draft, rather than directly from his client.
On 20 March 2024, the judge issued a written decision declining to impose Rule 11 sanctions on the lawyer. Although the court found that he had been negligent in failing to verify the citations, it did not find that he had acted in bad faith. The judge also declined to sanction the client, accepting that there was no basis to doubt his explanation that he believed the cases were genuine. There were no financial penalties, but the incident became part of the public court record and added to growing concerns about fabricated legal authorities appearing in court submissions.
What an auditable version would have shown
According to the court’s decision and the client’s affidavit, the fabricated citations began with a misunderstanding about what Bard was doing. The client believed he was searching for existing court decisions, when the chatbot had instead produced information that appeared to be legitimate legal research. Because the results included case names and citations in a familiar legal format, he had no apparent reason, based on his understanding of the technology, to suspect that the decisions were not genuine.
An auditable record of the interaction could have helped establish exactly what the client asked Bard, what the system returned and whether the citations were supported by identifiable legal sources. The court record does not establish what technical logs Google retained or whether the original conversation could have been recovered, but the absence of a verified source for each citation became important once the judge asked the lawyer to produce the decisions.
A separate record at the law firm would have addressed the next stage of the problem. Before the motion was submitted, each citation could have been checked against an authoritative legal database, with the result recorded alongside the filing. Such a record would have shown whether the cases had been verified, whether any searches had failed and who was responsible for reviewing the results before the document reached the court.
The distinction matters because the client and the lawyer made different mistakes. The client misunderstood the information Bard had generated, while the lawyer failed to verify the legal authorities he was relying on. A record of both stages would have helped establish how the fabricated citations moved from a chatbot conversation into a federal court filing, without depending entirely on later explanations of what each person believed had happened.
Where the gap was
The court’s decision exposed a failure that began outside the law firm and continued through its ordinary filing process. The client had used a general-purpose AI chatbot to conduct legal research, believing it would return genuine decisions in much the same way as a search engine. Although Bard could produce convincing legal language and citations, its responses were not proof that the cases existed, and the client did not recognise the difference between generated information and independently verified legal authorities.
The second failure occurred when the citations reached the lawyer. Rather than checking them against an authoritative source, he relied on his understanding of where the information had come from and incorporated the cases into a court filing. The judge found this failure negligent, even though the circumstances did not justify a finding of bad faith or the imposition of sanctions.
Headlights identifies a rules-behaviour gap in the lawyer’s failure to carry out an established professional responsibility. Lawyers are expected to verify the authorities they present to a court, regardless of whether those authorities come from a client, another lawyer or an AI system. The obligation had not changed, but the verification step was missed because the lawyer relied on an assumption about the source of the information.
The incident also illustrates automation bias, although the behaviour was different from the earlier Mata v. Avianca case. Here, the client placed confidence in Bard’s output because it resembled the results of a legal search, while the lawyer relied on citations he believed had already been identified by someone else. Neither independently established that the decisions existed before the motion was filed.
There was also a diffusion of responsibility across the process. The client believed the chatbot had supplied genuine cases, while the lawyer believed the citations had come through another legal professional. The court ultimately examined the conduct of each person separately, finding negligence on the lawyer’s part without concluding that either had knowingly submitted fabricated authorities.
These behavioural classifications are Headlights’ analysis of the documented conduct, not findings made by the court.
What governance should have looked like
The court’s decision demonstrated why legal citations need to be checked before they are submitted, regardless of how convincing they appear or who originally supplied them. A practical governance process would require every case cited in a court filing to be verified against an authoritative legal source, with the result recorded before the document could be approved for submission.
A CitationVerifier could perform that check by comparing citations with identifiable court decisions and flagging those that could not be confirmed. A VerificationGate could then require unresolved citations to be reviewed before a filing progressed, while a ConductRecord could preserve the verification results, the sources consulted and the person responsible for approving the document.
The process would also benefit from recording whether AI had been used to generate or identify legal authorities. In this case, the client did not tell his lawyer that Bard was the source of the citations, and the lawyer proceeded on a different understanding of where they had come from. Recording the origin of research material would not remove the lawyer’s duty to verify it, but it could help identify information that required particular attention.
These controls would not prevent every error, nor would they remove the need for professional judgement. They would, however, make it harder for an unverified citation to move from a chatbot response into a court filing without anyone establishing whether the decision existed.
Failure Pattern: a client obtained fabricated legal citations from Google Bard and passed them to his lawyer, who included them in a federal court filing without independently verifying them. The court found the lawyer negligent but declined to impose sanctions.
Governance Principle: legal authorities must be independently verified before they are submitted to court, regardless of whether they originate from AI, a client or another professional. Organisations should be able to demonstrate that verification took place and identify who approved the material.
The reference implementation of CitationVerifier, VerificationGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Decision and Order, United States v. Cohen, 1:18-cr-00602 (SDNY, 20 March 2024)
- Cohen affidavit explaining Bard use (29 December 2023)
- Judge won’t sanction Michael Cohen over AI-generated fake legal cases (Courthouse News)
- Fake citations in legal brief were generated by Google Bard AI program, says ex-Trump lawyer Michael Cohen (ABA Journal)
- Seemingly Hallucinated Cases, in Michael Cohen Post-Conviction Motion (Reason / Volokh)