What happened
Between July 2016 and November 2019, the Australian federal government raised debts against around 443,000 welfare recipients through a program officially called the Online Compliance Intervention. It became known as Robodebt. Around $1.76 billion of those debts were later found to be unlawful.
The method was a single piece of arithmetic. Each recipient’s annual income, as recorded by the Australian Tax Office, was divided by 26 to produce an estimated fortnightly income. That estimate was compared against the fortnightly income the recipient had actually declared while receiving payments. Any discrepancy was treated as an overpayment. A debt was generated and a notice sent automatically. No human reviewed the calculation before it went out, and no officer considered whether the income-averaging assumption was valid for that person’s actual employment pattern, seasonal work, casual shifts, periods of unemployment between jobs, multiple short stints with different employers.
For anyone who worked irregular hours, the averaging method invented debts that had never existed. A person who earned $26,000 across a year, $20,000 of it during four months when they were off welfare and $6,000 spread thinly across the rest, would be flagged for “underdeclaring” fortnightly income during the months they were on payments. The algorithm did not see the difference between an annual average and a fortnightly reality. The system did not check. The debt was raised.
People received letters demanding tens of thousands of dollars they did not owe. Debt collectors were engaged. Some recipients took out loans to pay debts that had been fabricated by the calculation. The Commonwealth sent threatening letters on behalf of the Department. The population most affected, people on Newstart, JobSeeker, single-parent payments, disability support, was the population least likely to know they had a right to dispute. The Royal Commission later found that a wide range of officials were aware of the scheme’s illegality and inaccuracy, and so of its potential to harm vulnerable recipients, and that warnings were repeatedly disregarded while the program continued.
On 27 November 2019, in the test case Amato v Commonwealth, the Commonwealth conceded that income averaging was not a lawful basis for raising debts. The scheme was wound down. On 29 May 2020, the Government announced that approximately 470,000 wrongly issued debts would be repaid, at a then-estimated cost of $721 million. A class action followed.
On 11 June 2021, in Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634, Justice Bernard Murphy of the Federal Court approved a settlement of $112 million in cash payments to group members, on top of debt waivers and refunds already underway. In his judgment, Justice Murphy described the proceeding as having “exposed a shameful chapter in the administration of the Commonwealth social security system and a massive failure of public administration.” Across cash, waivers and refunds, the Government’s total cost of unwinding the scheme exceeded $1.8 billion.
In August 2022 the Albanese government established the Royal Commission into the Robodebt Scheme. Its final report, delivered on 7 July 2023, ran to 990 pages across three volumes. Commissioner Catherine Holmes found that the scheme had been launched and continued despite repeated legal advice that it was unlawful, that senior public servants and ministers had misled the public about its operation, and that the design had been driven by budget projections rather than legal compliance. A sealed chapter referred multiple named individuals to authorities for civil, criminal and professional regulatory action.
What an auditable version would have shown
Robodebt did not use a large language model. The “algorithm” was a piece of arithmetic applied at population scale with no individual review. But the pattern is the pattern this library is about: an automated decision system produced a high-stakes output, a debt notice carrying legal force, and, when challenged, could not produce a verifiable record of how it had arrived at that output.
When recipients disputed debts, the Department’s response was to demand they prove the income averaging was wrong. Pay slips from years earlier. Employer records that had been destroyed under standard retention. Bank statements from accounts that had been closed. The burden was on the recipient. The algorithm itself produced no detail about its own reasoning, in part because no system requirement said it had to. There was no document the recipient could point at and say this is the calculation that led to this debt. There was just the debt.
An auditable conduct record would have produced something the recipient could read and a court could verify. For each debt: the input data used, with timestamps and ATO source identifiers; the averaging method applied, with the assumption that fortnightly income was uniform across the year captured as an explicit choice; a signed, tamper-evident record at the moment of generation, naming the ruleset version and the individual case officer who authorised the decision, or recording the absence of one in plain language. The reasoning made available on request, in English a welfare recipient could understand.
With that record, the conversation between the Department and a disputing recipient changes shape. The recipient sees what the algorithm assumed, what data it used, and which step is wrong. The Department sees whether the case ever passed a human. A court, the Ombudsman, the Auditor-General or a Royal Commission can audit the system end to end without taking the Department’s word for what happened. None of this existed.
Where the gap was
The gap was not a software defect. It was a policy decision, implemented in software, to treat algorithmic outputs as authoritative without an audit trail anyone could inspect.
The Robodebt design was specifically configured to remove human review. Before 2016, debt notices were prepared by departmental officers who used income averaging only when more granular data was unavailable, and who were trained to apply judgment. The 2016 program flipped that default. Debts were raised automatically and a human reviewer was only required if the recipient initiated a dispute. Because the affected population, people in financial distress, often unwell, often without legal representation, was the population least likely to know they had a right to dispute, the human-in-the-loop step was effectively removed by design.
There was no system requirement to keep a per-debt record of the calculation, the input data, or the absence of human review. The Department could later say, in aggregate, how many debts it had issued. It could not say, for any individual debt, who had decided the income-averaging method was appropriate for that person’s employment pattern. The answer was: no one had, because the system was configured so that no one had to.
What governance should have looked like
Every algorithmic decision that imposes a legal obligation on a person, a debt, a benefit reduction, a denial, a sanction, is written to a signed conduct record at the moment of generation. The record names the rule applied, the data sources consulted, the version of the ruleset that was active and the human officer who authorised the decision. If no human officer authorised it, the record says so explicitly, and the system is configured to refuse to release any output that imposes a legal obligation without that authorisation.
The controls Robodebt would have needed all involved producing records that someone outside the Department could read.
A constraint that refused to release any debt-imposing output without a named officer’s authorisation on the individual case. Such a control would have collapsed the program back to manual scale. It would have surfaced, at the system level, the policy choice the Department had made to remove human review.
A conduct record on every debt. Plain-English explanation. Input data. Rule applied. Officer name. With that record, a person disputing the debt could see what the system had assumed, point at the wrong step, and have the dispute resolved on facts. Without it, the recipient had no document to argue from and the Department had no document to defend with.
Independent auditability. The Royal Commission found the Department had concealed legal advice that the scheme was unlawful, repeatedly, for years. An independent auditor, the Ombudsman, the Commonwealth Auditor-General, a journalist with FOI rights, would have caught this pattern earlier if a verifiable, tamper-evident record had existed across the program. Hundreds of thousands of debts. No individual human authorisation. The same automated explanation, repeated. There would have been no plausible deniability.
Robodebt is the foundational Australian case study in what happens when automated decisions are released at scale without an audit trail anyone outside the issuing agency can read and verify. The fact that the underlying calculation was arithmetic rather than a neural network is incidental. The pattern is the pattern AI agents are now reproducing across denials, sanctions, terminations and charges. The question for the next program is whether it is set up, on day one, to produce records anyone outside the agency can inspect.
The reference implementation of ConductRecord and MetricRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. Anyone can read every line and verify the signatures. The repository is public now.
Sources
- Royal Commission into the Robodebt Scheme, Final Report
- Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634
- Federal Court approves $112 million Robodebt settlement (Human Rights Law Centre)
- Robodebt scheme (Wikipedia)
- Robodebt Royal Commission: ministers and officials referred for civil and criminal action (ABC News)
- Explainer, Deanna Amato’s robo-debt case (Victoria Legal Aid)