The biggest losses in software history were, with one deliberate exception, not attacks. They were silent, correlated, self-inflicted — and they teach the exact risk autonomous AI agents are about to make expensive again.
At 9:30 in the morning on August 1, 2012, Knight Capital Group was one of the largest trading firms in the United States, executing a sixth of all the volume on the New York Stock Exchange. By 10:15 it was, for practical purposes, finished. In those forty-five minutes a piece of its own trading software (not a hacker's, its own) fired more than four million unwanted orders into the market, accumulating roughly $7 billion in positions the firm never meant to hold and a loss of about $440 million by the time humans understood what their machine was doing. The cause, documented in the SEC's administrative proceeding, was almost insultingly small: a deployment that updated seven of eight servers. The eighth still carried a dormant piece of code called Power Peg, retired years earlier, and the new release reused the old feature flag that woke it up.
No one attacked Knight Capital. The market data was accurate, the exchange functioned perfectly, and every system reported itself healthy while the company bled ten million dollars a minute. That shape (no adversary, no alarm, one change propagating everywhere at once) turns out to be the shape of almost every entry on the list below.
We've written before about the biggest bug-bounty payouts in history, the ledger of what it costs when someone does attack. This is the other ledger, the bigger one: what software has cost when nobody attacked at all.
Every figure below states what it counts, and comes from a primary or authoritative source (inquiry boards, SEC filings, statutory inquiries) linked at the end.
1. CrowdStrike outage (2024) — roughly $5.4 billion in direct losses to Fortune 500 companies alone (estimate). One faulty content update to the Falcon Sensor security agent blue-screened Windows machines worldwide: airlines grounded, hospitals postponed procedures, broadcasters went dark. The $5.4 billion figure is insurer Parametrix's estimate of direct losses to the Fortune 500 excluding Microsoft. It's an estimate, not an audited total, and worth quoting with its second half: Parametrix put the insured portion at only $0.54–1.08 billion. Delta Air Lines' CEO alone put his company's cost at about $500 million. A security product built to stop attackers did more single-day damage than most attackers ever have, via an update everyone trusted enough to install automatically.
2. NotPetya (2017) — ~$10 billion in global damages; Merck's property insurance claim alone ~$1.4 billion. The one genuine attack on this list, included to mark the contrast. A wiper delivered through the auto-update channel of Ukrainian accounting software spread worldwide in hours; Merck, Maersk, FedEx's TNT and others took nine- and ten-figure hits. Even here, the biggest financial story wasn't the malware's cleverness. It was that the losses landed in a coverage category nobody had priced. Merck's insurers argued war exclusions; the industry ended up naming the gap "silent cyber." The attack was loud. The exposure had been silent for years.
3. Boeing 737 MAX / MCAS (2018–2019) — 346 deaths; over $20 billion in direct costs. Two crashes, Lion Air 610 and Ethiopian Airlines 302, killed everyone aboard. MCAS, a flight-control augmentation system, repeatedly commanded the nose down based on a single failed angle-of-attack sensor: software acting confidently on one bad input, with authority the crews struggled to override. The direct bill to Boeing (fines including a $2.5 billion DOJ settlement, victim compensation, airline compensation, legal costs) passed $20 billion, with tens of billions more in cancelled orders. The deadliest entry here, and the clearest case of automation trusted past its evidence.
4. Post Office Horizon (UK, 1999–2015) — about £1.48 billion paid in redress so far; more than 900 subpostmasters wrongfully convicted. Fujitsu's Horizon accounting system generated phantom shortfalls in branch accounts. The Post Office prosecuted its own subpostmasters for theft and false accounting on the system's say-so. People went to prison, went bankrupt, and died without seeing their names cleared, while the institution treated the software's numbers as more credible than its humans. The statutory inquiry and Parliament put the redress bill around £1.48 billion and climbing. Nobody hacked Horizon. It was simply wrong, quietly, for years, and trusted.
5. Zillow Offers (2021) — a write-down north of $500 million and the end of the business line. Zillow's iBuying arm bought homes at prices its Zestimate-driven models predicted it could resell profitably. In volatile 2021 markets the algorithm systematically overpaid, not on one house, on the whole book. Zillow's own Q3 2021 SEC filing recorded the inventory write-down; the company shut the unit and laid off about a quarter of its staff. No bug, in the classic sense: the code did exactly what it was designed to do. The model was just confidently wrong at portfolio scale, and it was allowed to act.
6. Knight Capital (2012) — ~$440 million lost in ~45 minutes. The opening scene above: one incomplete deploy, one reused flag, one dormant code path, and an automated system executing its mistake four million times faster than anyone could stop it.
7. Ariane 5 Flight 501 (1996) — roughly $370 million of rocket and payload, gone 40 seconds after liftoff. Europe's new heavy launcher self-destructed on its maiden flight. The ESA inquiry board traced it to a floating-point value converted to a 16-bit integer in code inherited from Ariane 4, a routine that wasn't even needed after launch on the new rocket. The conversion overflowed, the exception went unhandled, both inertial reference units shut down, and the vehicle's flight computer obediently steered into breakup. Dead code from the last system, fatal on the next one.
8. Mars Climate Orbiter (1999) — a $327.6 million mission lost to a unit conversion. That figure, from NASA's mishap board, is the whole mission (orbiter, lander, and operations) not just the spacecraft. Lockheed Martin's ground software reported thruster impulse in pound-force seconds; NASA's navigation software read the numbers as newton seconds. Every trajectory correction was subtly wrong in the same direction, and the orbiter hit the Martian atmosphere instead of orbiting above it. Two teams, two conventions, one interface nobody checked, and months of green dashboards while the error compounded.
9. AT&T long-distance collapse (January 15, 1990) — nine hours, about half of AT&T's long-distance calls failing, roughly $60 million in lost call revenue plus knock-on chaos (500 delayed flights). A one-line error, a misplaced break in C, in the recovery software of the 4ESS switches. One switch reset itself; its "I'm back" message arrived at neighbors at exactly the wrong moment and crashed them; their recoveries crashed others. All 114 switches ran the same code, so the network tore itself down in a wave, over and over, for nine hours. The upgrade had been tested. The failure only existed at network scale.
10. Therac-25 (1985–1987) — six massive radiation overdoses, three of them fatal. The canonical safety-critical case, documented in Leveson and Turner's investigation. A race condition in a radiation-therapy machine's control software: an operator who edited settings quickly could fire the high-current electron beam without its X-ray target in place, delivering doses up to two orders of magnitude beyond therapeutic levels. Earlier models had hardware interlocks that made this physically impossible; the Therac-25 removed them and trusted the software. The smallest dollar figure on this list, and the reason "the software checks it" stopped being an acceptable safety argument.
Put the list side by side and one fact stands out: the biggest losses in software history were, with one deliberate exception, not attacks.
A deploy that skipped a server. A unit that didn't match. Dead code from the previous rocket. A break statement in recovery logic. A pricing model that was wrong about a whole market. An update everyone trusted. Even NotPetya, the one true attack, does most of its work in this frame: the unpriced exposure it detonated had been sitting silently in policy language for years.
Three traits repeat, and they're worth naming precisely, because they are the opposite of how most people picture a software catastrophe:
1. Non-adversarial. There is no villain to stop at the perimeter. The threat model that dominates security budgets, a hostile outsider, describes almost none of the damage on this list. The system's own operators, vendors, and automation did this.
2. Silent. None of these announced themselves. Knight's dashboards were green while it hemorrhaged; Zillow's model looked fine as it overpaid; Horizon's numbers were so plausible that courts convicted on them; Mars Climate Orbiter flew a subtly wrong course for months. The failure mode was not a crash. It was confident wrong action: the system doing something, smoothly, that no one wanted done.
3. Correlated. One upstream change hit everything that shared it, at once. One content file bricked millions of machines; one switch image took down 114 switches; one inherited module ended a rocket; one model mispriced an entire inventory. Software's superpower, perfect replication, is precisely what makes its failures common-mode. There is no idiosyncratic bad luck here; there is one mistake, everywhere, simultaneously.
We've made the argument before, in Inside the Uninsured Middle and Correlated Risk, By Stack, that autonomous AI agents create an operational risk class the market hasn't priced: not "what if it's hacked" but "what if it's wrong, at speed, everywhere." This list is that argument's fossil record. We didn't invent the loss-shape; thirty years of incident reports did, and the insurance industry is already circling it.
Because look at what an AI agent is, in the terms this list teaches: software that acts on its own authority, continuously, against live systems, on a model's judgment. Knight Capital was one fixed algorithm acting autonomously for forty-five minutes, and it produced the fastest nine-figure loss in this catalog. An agent fleet is that premise generalized: the trusted update channel, the confidently wrong model, and the correlated deployment, all in one artifact, with a natural-language attack-and-error surface on top.
The pattern says the next entries on this list won't look like break-ins either. They'll look like Zillow's quarter: the system did what it was designed to do, the design was wrong about the world, and it acted anyway, silently, in parallel, everywhere it was installed. The organizations that fare best will be the ones that learned the old lessons before the new tooling made them expensive again: stage deploys and verify all eight servers; distrust any single sensor; put hardware-grade interlocks around software-grade confidence; check the units at every interface; and when the machine's numbers and your people disagree, remember Horizon before you side with the machine.
Knight Capital's engineers, by every account, were competent people at a serious firm. So were NASA's, ESA's, AT&T's, Boeing's. The lesson of the most expensive software failures in history is not that someone was careless. It's that trusted automation fails differently than people do, without malice, without noise, and all at once, and that the bill for forgetting this has been arriving, regularly, for forty years. The only thing that changes is the number of zeros.
Every failure on this list was non-adversarial, silent, and correlated. That is the exact shape of autonomous-agent risk — and it is the shape you can build against.
Stage the deploy and verify all eight servers, gate the irreversible action behind a human, keep a tamper-evident record of what the system actually did, and instrument for the silent stop rather than the crash. The Agent Trust Stack is the harness that makes those disciplines answerable for an agent deployment, and Chain of Consciousness is the append-only, anchored record that lets you tell "working" from "confidently wrong" — the difference Horizon's courts could not.
See Hosted Chain of Consciousness · Read the Theory of Agent Trust
pip install agent-trust-stack · npm install agent-trust-stack