Take a number from a tax return. Divide it by twenty-six. Treat the result as a fact about every fortnight of a person's year. Then bill them for the difference between that fact and what they reported at the time, and put the burden of disproving your arithmetic on them.
That is the entire technical content of Robodebt, the Australian government's automated welfare-debt scheme, which ran from 2015 to 2019, raised hundreds of thousands of unlawful debts against people on part-rate benefits, and ended in a Royal Commission whose final verdict has the cadence of a sentence handed down: Robodebt was "a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals." Commissioner Catherine Holmes delivered that report on 7 July 2023, in three volumes, nine hundred pages, with fifty-seven recommendations.
Engineers should study this case the way pilots study crashes, and not for the reason usually given. The usual reading, the one you have probably encountered, is that Robodebt is what happens when you take the human out of the loop. That reading is true, it has been standard since about 2020, and it stops one layer short of the interesting part. Because the record assembled by the Royal Commission supports something more precise and more uncomfortable: the humans were not lost in an automation project. Removing them was the business case. The projected savings were, almost exactly, a measurement of how much error-correction the humans had been doing.
The headline figure long attached to Robodebt is 1.8 billion Australian dollars. As of June 2026 the running total across both settlements is more than 2.4 billion, and it is worth decomposing, because the decomposition is more damning than either round number.
In 2021, in the class-action settlement approved in Prygodicz v Commonwealth, the Federal Court signed off on a payment of about AU$112 million, covering interest and legal costs. The rest of the package was the Commonwealth unwinding its own scheme: roughly AU$751 million in refunds of money it had already collected, and about AU$1.76 billion in asserted debts wiped for roughly 433,000 people, most of which had simply never existed as lawful obligations. In May 2020 the government had announced it would repay 470,000 wrongly issued debts worth about $720 million. Keep the units straight, because the record does: 470,000 is a count of debts, not of people; the class action had about 648,000 group members.
The decomposition did not stop moving after 2021. On 23 June 2026, Justice Beach approved a second settlement in the same proceeding: about AU$548.5 million more, of which roughly AU$475 million is compensation for about 125,000 registered group members, alongside up to AU$60 million to administer the distribution scheme and AU$13.5 million in legal costs, all paid by the Commonwealth. That further amount is, by itself, the largest class-action settlement in Australian legal history, and it sits on top of the AU$112 million compensation component approved in 2021. Gordon Legal, who ran the class action and now administer the scheme, put the total across both settlements at more than AU$2.4 billion; registration closed in May 2026, and payments are anticipated between October 2026 and February 2027.
Read that decomposition again with an engineer's eye, and keep it honest across both settlements. Compensation flowing to victims, even after June 2026, is roughly AU$587 million of a total north of AU$2.4 billion: a bit under a quarter. The rest was the state deleting invoices that its own arithmetic had fabricated and refunding money it should never have collected. The 2026 settlement moves real remedy to real people, and the proportions still say what they said in 2021: the money mostly measures the size of the error, not the size of the remedy.
Averaging an annual income over twenty-six fortnights is not inherently absurd. For a salaried worker with stable pay, it is even correct. The failure is sharper than "averaging is imprecise," and Peter Whiteford, a social-policy professor at the Australian National University who analysed the Department of Social Services' own data, stated it plainly in 2023: the data "shows almost nobody who received income had completely stable earnings" during the Robodebt period, and significant numbers of recipients were on payments for only part of any financial year.
Almost nobody. The people in this system were on part-rate benefits precisely because their work was casual, seasonal, irregular: a few shifts one fortnight, none the next, a burst of retail hours in December. Annual-divided-by-26 is a model whose one load-bearing assumption is that income is flat across the year, deployed on a population that qualifies for the system by having income that is not flat across the year. The selection criterion for entering the pipeline was the same property that invalidated the pipeline's model. When you hear that shape, save it: a cohort selected for variance, modelled with a statistic that assumes none. It has cousins everywhere. The churn model trained on your most stable customers. The latency budget set from average load, applied to the tail that only exists because load is not average.
Whiteford adds the design irony that deserves its own line: Australian social policy had spent four decades, since 1980, deliberately encouraging benefit recipients to take part-time and casual work. The averaging model punished exactly the behaviour the policy existed to produce.
Before Robodebt, income averaging existed in the system, and here is where the standard human-in-the-loop story earns its paragraph. The Royal Commission's record shows averaging had been used relatively seldom, usually by agreement with the recipient, and in the context of other information. A caseworker, unable to establish actual fortnightly earnings any other way, might average as a last resort, with a person on the other end of the process and other evidence in the file. The automation did not invent the formula. It deleted everything around the formula: the human, the agreement, the context, the last-resort status. Roughly 470,000 times.
All true. Now go one layer down, because the Commission's record does.
Inside the Department of Human Services, as the scheme was being designed, an official articulated something remarkable, preserved in the Royal Commission's account of the scheme's design: to exclude automatic, default averaging in favour of manual investigation, with averaging only as a last resort, "would render OCI physically and economically untenable." OCI was the Online Compliance Intervention, the scheme's official name.
Read it twice, because it inverts the usual story of automation gone wrong. This is not a team that automated a process and failed to notice a safeguard falling away. This is a written acknowledgment that the scheme was only economically viable if the verification step was removed. Manual investigation was not overhead that the computer happened to make redundant. Its removal was the product. The efficiency being sold to cabinet was, in nearly its entirety, the cost of checking whether the debts were real, and the projected savings were therefore a rough measurement of how much error-correction the verification layer had been performing all along.
That is the sentence to carry back to your own systems. When a proposal's savings come from removing a review step, the honest first question is not "will the automation be accurate?" It is "what was the review step catching, and did we ever measure it?" A working error-correction layer is invisible in the ledger precisely when it is working; its value only becomes legible as a catastrophe after it is gone. Robodebt is what the invoice for that invisibility looks like at national scale.
In November 2014, before the scheme scaled, the Department of Social Services, the department that owned the legislation, was advised that income averaging as proposed "did not accord with legislation." The record then adds the qualifier an honest telling has to keep: there is no evidence that this advice reached the Department of Human Services, the department building the scheme, at that time.
Resist the flat version, "they knew it was illegal and shipped it anyway," because the documented version is a more familiar engineering story and a more useful one: the finding existed, in writing, in the organisation that owned the rules, more than a year before rollout, and it failed to cross the boundary to the organisation that owned the build. Anyone who has watched a security review's findings fail to reach the team with the deploy keys knows this failure mode personally. A known defect that cannot traverse an org chart is functionally an unknown defect, except that afterward, in the inquiry, it looks much worse.
What happened later, once the scheme was live and questioned, moved from failure to concealment: the Royal Commission found that officers of both departments "engaged in behaviour designed to mislead and impede the Ombudsman" during the 2017 investigation that could have stopped the scheme years earlier.
Nearly everything written about Robodebt ends with the Royal Commission in July 2023. The story did not end there. The Commission made confidential referrals; the National Anti-Corruption Commission initially declined to pursue them, a decision that was itself reviewed by the NACC's Inspector, and, after private hearings with the six referred individuals and thirty-three witnesses, the NACC published its investigation report in March 2026.
Its findings, as published and reported: two of the six engaged in serious corrupt conduct. Mark Withnell, formerly the Department of Human Services' general manager of business integrity, was found to have intentionally misled officers of the Department of Social Services in 2015 in the preparation of the submission that took the scheme's proposal to the Expenditure Review Committee of Cabinet. Serena Wilson, formerly a deputy secretary at Social Services, was found to have intentionally misled the Ombudsman in 2017, having concealed legal advice that the scheme was unlawful. The NACC found insufficient admissible evidence to refer either for prosecution, and it made no corruption finding against Scott Morrison, the former prime minister who had been social services minister when the scheme was conceived.
Put the Withnell finding beside the economics of the previous section and hold them together, because they are one mechanism seen twice. The savings came from deleting the verification step; the submission that sold those savings to cabinet was found, a decade later, to have been intentionally misleading. The business case and the corruption finding are about the same document. Systems that are only viable without checking tend to be sold by processes that are also not checking, and the record now says so with names attached.
Robodebt gives engineers four durable tests, each purchasable for far less than AU$2.4 billion.
When savings are the pitch, locate the deleted step. If an efficiency case rests on removing review, verification, appeal, or reconciliation, then the projected saving approximately equals the value of the checking you are about to stop doing. Demand the measurement: what does that layer currently catch, at what rate, at what severity? If nobody can answer, the layer's output has never been observed, which means the savings estimate is a guess about a control nobody understood.
Check the model's premise against the population's selection rule. Ask of any pipeline: is the property that routes people or records into this system correlated with the property my model assumes away? A system for irregular earners that assumes regular earnings is not an approximation. It is a category error with decimal places.
Trace the defect's path across boundaries, not just its existence. The November 2014 advice existed. The question that mattered was whether it could travel from the department that owned the law to the department that owned the code. Your equivalent: does a finding logged by legal, security, or compliance mechanically reach the team that ships, or does it depend on someone forwarding an email?
And when the loop matters, name which loop. "Human in the loop" was never the precise safeguard here; the precise safeguard was manual investigation before a debt was asserted, and the humans elsewhere in the system did not compensate for its absence. If your safety story says "a human reviews it," ask which human, reviewing what, empowered to stop what, and what happens to throughput when they do. Robodebt's designers could answer that last question exactly. That was the problem.
Sources: Royal Commission into the Robodebt Scheme, final report (Commissioner Catherine Holmes AC SC, 7 July 2023), quoted findings as reproduced by the Law Society Journal (Aug 2023) and the Commission's published overview; Whiteford, "Income averaging key source of mistaken robodebt debts," ANU (Mar 2023); Prygodicz v Commonwealth (No 2) [2021] FCA 634 settlement reporting (AU$112M approved; ~AU$1.76B debts wiped; ~AU$751M refunds); SBS News on the May 2020 repayment of 470,000 debts (~$720M); National Anti-Corruption Commission, investigation report into the Robodebt referrals (March 2026), as published by the NACC and reported by ABC News, Monash Lens, and The Conversation; Gordon Legal, Robodebt class action appeal settlement page (approval by Beach J, 23 June 2026: ~AU$548.5M comprising ~AU$475M compensation, up to AU$60M administration, AU$13.5M legal costs; totals across both settlements; registration and payment dates), with the approval also reported by SBS News (June 2026).
If an efficiency case in your stack rests on removing a check, the essay's first test needs an answer somebody wrote down: what was that layer catching, at what rate, at what severity? A working error-correction layer is invisible in the ledger precisely when it is working, and its value only becomes legible after it is gone. Chain of Consciousness records the reasoning behind a decision as a durable artifact, so the question can be answered from the record instead of reconstructed after the invoice arrives.
pip install chain-of-consciousness
npm install chain-of-consciousness