What happened
The ABC reported that in 2024 Australian Catholic University recorded nearly 6,000 academic-misconduct referrals across its campuses, and that about 90 percent related to suspected AI use, a figure ACU said was substantially overstated. ACU had adopted Turnitin’s AI-writing indicator in 2023; the tool highlights suspected AI-generated text in blue and returns a percentage estimate of how much of a submission it judges to be machine-written. According to that reporting, internal documentation indicated the university had regularly relied on the indicator, at times as the main basis for a referral. Turnitin’s own guidance cautions that its reports “may not always be accurate” and should not be the sole basis for sanctions, and independent evaluations have found AI detectors unreliable under realistic conditions, with no tool reliably accurate across tasks. ACU said that any case in which Turnitin was the sole evidence was dismissed.
Students reportedly described months in limbo. One final-year nursing student, Madeleine, reportedly had her results marked “withheld” for six months after an essay was flagged, which she says contributed to her not being offered a graduate position she needed to register. A paramedic student told the ABC that almost his entire essay “was lit up in blue, 84 per cent of it supposedly written by AI.” Some students were reportedly asked to produce search histories and drafts to prove their innocence, reversing the burden of proof onto the accused. Internal records reportedly show ACU was aware of the tool’s unreliability for more than a year before withdrawing it in March 2025, and on review roughly a quarter of the referrals were dismissed.
What an auditable version would have shown
A probabilistic score is a lead, not a finding. An auditable version records, for each allegation, what the detector actually returned, what corroborating evidence existed beyond it, and who decided to proceed on what basis, signed when the decision was made. In aggregate it would have surfaced, early, how often referrals rested on the score alone and the false-positive exposure that implied, while that was still a warning rather than a controversy touching thousands of students.
Where the gap was
The gap was treating a model’s estimate as verification. A detector reports that a text “looks AI-generated”; it does not establish that it is, and the vendor says as much. A VerificationGate is the control: a flag is a proposal to be checked against real evidence (process history, drafts, an oral defence), never an adverse finding on the model’s say-so, and it routes the question to a trusted source rather than back to another model. A MetricRecord over signed records would have exposed how often the indicator was the sole basis, and the dismissal rate, while they still mattered. A ConductRecord ties each case to the evidence it actually stood on, so a referral resting only on the detector is visible in the file, not discovered on appeal.
What governance should have looked like
An accusation that can withhold a degree needs more than a number from a tool whose own vendor says not to rely on it alone. Corroboration required before an allegation is raised, the burden of proof kept where it belongs, and the reliability of the instrument monitored continuously against real outcomes: those are the ordinary safeguards of any high-stakes decision. Used as one signal among several, an AI indicator is a prompt to look closer; used as the finding, it turns a probability into a verdict, and that is how thousands of students were drawn into a misconduct process on the strength of a score.
The reference implementation of VerificationGate, MetricRecord, and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- University Using AI to Falsely Accuse Students of Cheating With AI (Futurism, reporting ABC News Australia)
- A university decided to use an AI detector to detect AI cheating. Then it had two problems. (Boing Boing)
- When the detector becomes the accuser: why universities must rethink AI policing (CAQA, citing Weber-Wulff et al.)