A senior Melbourne barrister filed submissions in a Supreme Court homicide hearing that cited cases the AI had invented, including quotations attributed to the judge himself

What happened

In August 2024, Rishi Nathwani KC, a senior member of the Victorian Bar, was leading the defence in a Supreme Court of Victoria homicide matter. The accused was seventeen years old. The proceeding was a hearing on legal questions before trial.

In the written submissions filed for that hearing, Nathwani's team cited a series of authorities to support a particular legal argument. When the court turned to those authorities, neither the judge's associate nor the prosecution team could find them. Some of the case names did not exist. Some of the case names existed but did not say what the submissions claimed they said. The submissions also contained fabricated extracts purporting to quote a speech to the Victorian Parliament that had never been given in the form quoted.

The hearing was adjourned. When the court reconvened the following day, Nathwani told Justice Elliott that an AI tool had been used in preparing the submissions, that the citations had not been verified before filing, and that he apologised to the court for what had occurred. Justice Elliott referred the matter to the Victorian Legal Services Board for disciplinary consideration. The criminal proceeding continued. The defendant was not prejudiced by the adjournment.

The case was widely covered in the Australian press. It became the most senior Australian legal practitioner publicly known to have submitted hallucinated AI-generated authorities to a court, and prompted the Supreme Court of Victoria and several state Bar associations to issue guidelines on the use of generative AI in legal practice.

What an auditable version would have shown

Nathwani's chambers, like almost every set of Australian chambers in mid-2024, had no audit trail attached to AI-assisted legal research. The barrister or junior staff member who ran the queries could not, after the fact, produce the prompts used, the model and version called, the dates of the queries, or the verification steps (if any) that had been performed on the output before it reached the brief.

A conduct record built into the research workflow would have produced something different. Every AI query captured at the moment it was run, with the prompt, the model version, the date and time, and the response. Every citation extracted from the response logged separately, with a status field of verified, partially verified, or unverified, and the verification source where applicable. When the brief was finalised, a check would compare every citation in the document against the verification log. Any citation marked unverified would block the document from being filed.

With that record, the chambers would have known, before the document left their hands, that several authorities had not been confirmed against a primary source. The error would have been caught at the desk, not in open court.

Where the gap was

The legal profession in Australia, as everywhere else, was operating in 2024 on a thirty-year-old assumption. The assumption was that any case citation appearing in a senior counsel's brief had been verified by someone, somewhere, against an actual judgment. The assumption was sound when legal research was done in the books and on the established databases such as AustLII, Jade, LexisNexis, and Westlaw, because every search returned a real authority and the verification was implicit in the lookup. The assumption broke the moment a generative AI tool was introduced into the workflow, because generative AI returns plausible-looking authorities whether or not they exist.

There was no documented expectation in Nathwani's chambers, and very few documented expectations across the Australian profession at that point, that AI-generated research had to be verified against primary sources before use. The Bar's professional rules contained the general obligation that counsel must not mislead the court, but no procedure for ensuring that obligation survived the introduction of a tool that hallucinates. The gap was procedural, and it was widespread. Nathwani's case happened to be the first that broke into open court at the senior end of the profession. There were certainly others that did not.

What governance should have looked like

The defence is not, and should never have been, "don't use AI." The defence is a verification gate, built into the research workflow itself, that no document can pass without.

from headlights import CitationVerifier, ConductRecord, sign, chain
from datetime import datetime, timezone

# Each AI-assisted research session captured as a signed record
session = ConductRecord(
    agent_id="legal-research-assistant",
    operator="chambers-staff-anonymised-id",
    model_provider="anthropic",
    model_version="claude-3-5-sonnet-20240620",
    prompt=research_prompt,
    response=ai_response,
    timestamp=datetime.now(timezone.utc),
)
chain.append(sign(session, key=chambers_key))

# Every citation extracted from the response is verified before use
verifier = CitationVerifier(
    primary_sources=["austlii", "jade", "high-court-of-australia", "supreme-court-vic"],
    require_two_source_match=True,
)

for citation in extract_citations(ai_response):
    result = verifier.check(citation)
    # result.status is one of: VERIFIED, NOT_FOUND, QUOTE_MISMATCH, AMBIGUOUS
    record = ConductRecord(
        agent_id="citation-verifier",
        citation_text=citation.text,
        claimed_quote=citation.quote,
        primary_source_check=result.source_url,
        status=result.status,
        timestamp=datetime.now(timezone.utc),
    )
    chain.append(sign(record, key=chambers_key))

# Before any brief is filed, the brief is run through a final gate
brief_audit = verifier.audit_document(brief)
if brief_audit.unverified_citations:
    # The brief cannot be filed until each is verified or removed.
    raise FilingBlocked(brief_audit.unverified_citations)

The CitationVerifier module is one layer. The other is procedural. A chambers practice rule that any document containing an authority touched by an AI tool, including indirectly through summarisation or research support, must pass the verifier before filing, and that the verifier's signed log is retained for the duration of the chambers' professional indemnity exposure. That practice rule did not exist at the Victorian Bar in August 2024. Several state Bars have since published guidance moving towards it.

The third layer is the duty to the court. A practitioner who has used AI in the preparation of submissions should be expected to disclose that use to the court and to be able to produce, on request, the signed conduct record showing how the citations were verified. That disclosure is what restores the assumption that authorities in a brief have been checked by someone, somewhere, against a real judgment. The signed record is what makes the disclosure verifiable rather than performative.

The reference implementation of CitationVerifier is open source. It lives in the same repository as the rest of the Headlights primitives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed, free for any chambers, firm, or in-house team to install. The repository goes public alongside the launch of this Incident Library.