A quantum-supremacy claim built on a signal two-tenths of one percent above noise, erased in five years by ordinary GPUs, certified by a metric that turned out to be spoofable. It is the cleanest available preview of what is happening to AI leaderboards right now.
In October 2019, Google published a paper in Nature announcing that its 53-qubit Sycamore chip had performed, in about 200 seconds, a computation that would take the world's most powerful supercomputer roughly 10,000 years. The word “supremacy” was in the title. The press coverage wrote itself, and for a week the story was everywhere: quantum computers had crossed the line.
Two numbers from that paper became famous. Two hundred seconds. Ten thousand years.
A third number stayed obscure, and it was the one that decided how the story would end: 0.2 percent. That was the fidelity of the result, measured by the experiment's own verification metric. Put plainly, about 99.8 percent of what Sycamore produced was noise. The supremacy claim rested on detecting a signal two-tenths of one percent above random garbage, and the paper said so, openly, for anyone who read past the headline.
Five years later, the 10,000-year estimate was dead many times over, killed not by a rival quantum computer but by ordinary classical software running on GPUs. And the verification metric itself, the thing that had certified the whole claim, turned out to be spoofable.
If you build or buy AI systems, this story should feel uncomfortably familiar. Your benchmark scores are Sycamore's fidelity number wearing a different outfit. The anatomy of that claim, how it was constructed, why it eroded, and what finally discredited the instrument that certified it, is the cleanest available preview of what is happening right now to AI leaderboards.
Start with what Sycamore actually did, because the details are where the lesson lives.
The task was called random circuit sampling: run a random sequence of quantum gates, then sample bitstrings from the resulting probability distribution. It was chosen precisely because it is brutally hard for classical computers to simulate. It has no practical application. It exists to be hard.
Here is the problem: how do you check the answer? You cannot verify the output directly, because verifying it requires the same classical computation you claimed was impossible. So the Google team used a proxy called linear cross-entropy benchmarking, XEB, which measures how much the sampled distribution leans toward the ideal quantum one rather than uniform randomness. A perfect quantum computer scores 1.0. Pure noise scores 0.
Sycamore scored about 0.002 on its hardest circuit. That was enough, statistically, to say the chip was doing something classically expensive. But hold the shape of this in your head: the claim was certified by a proxy metric, at a margin barely above noise, on a task selected for difficulty rather than usefulness.
Now watch what happened to it.
Within days, IBM published a rebuttal arguing that Summit, the supercomputer in Google's comparison, could do the task in about 2.5 days rather than 10,000 years, using secondary storage the estimate had ignored. Not a new machine. A better accounting of an existing one. The gap shrank by six orders of magnitude before the champagne was flat.
In 2021, Feng Pan, Keyang Chen, and Pan Zhang simulated the Sycamore circuits classically on a cluster of 60 GPUs, producing a million bitstrings with an XEB fidelity of 0.739. Read that against the original: the classical “loser” reproduced the task at roughly 370 times the fidelity of the quantum chip that had claimed supremacy over it. By 2022, tensor-network methods had the job down to hours. By 2024, a distributed approach running on 1,432 NVIDIA A100s reported the equivalent sampling task in 86.4 seconds, faster than Sycamore's original 200.
From 10,000 years to under Sycamore's own runtime, in five years, without a single qubit involved. The erosion came entirely from unglamorous classical engineering: better tensor contractions, smarter memory use, more GPUs.
Then came the deepest cut. In 2024, researchers demonstrated that linear XEB itself can be classically spoofed: you can produce a good XEB score without performing the hard computation the score supposedly certifies. A companion result had already shown the same for boson-sampling benchmarks. The metric that anointed supremacy does not, by itself, distinguish a quantum computer from a sufficiently clever classical impostor.
None of this makes Sycamore fake. It was a genuine engineering landmark, and the physicists involved were upfront about the fidelity. What died was not the chip. It was the claim structure: a headline gap, certified by a proxy, at a sliver of a margin, on an instrument that could be gamed. That structure had a half-life, and every part of it decayed on schedule.
Now translate each piece into the language of AI evals, because the correspondence is not a metaphor. It is the same measurement problem with the same failure modes.
An AI benchmark is a proxy distribution. MMLU does not measure “understanding”; it measures accuracy on a fixed set of multiple-choice questions that we hope correlates with understanding. SWE-bench does not measure “software engineering”; it measures whether a patch passes certain tests on certain GitHub issues. Every benchmark is an XEB: a computable stand-in for a quantity you cannot check directly.
So the honest question is the fidelity question. Of the score, how much reflects the capability the leaderboard claims, and how much is noise, or worse, leakage?
The numbers we have are not reassuring. Analyses of pretraining corpora have found MMLU test items appearing nearly verbatim in Common Crawl, with contamination touching an estimated 5 to 10 percent of test questions in standard corpora. A model can earn those points by recall while the leaderboard calls it reasoning.
SWE-bench is starker. Audits have documented solution leakage affecting roughly a third of instances, 32.7 percent in one accounting, plus score inflation from weak test suites that pass wrong patches. And OpenAI's own evaluation work reported the purest possible symptom: models reproducing the gold patch, or the issue text itself, when given nothing but the task ID. Stop and sit with that one. When the answer is recoverable from the identifier of the question, the benchmark is measuring memory, not engineering. That is the AI world's 0.2 percent fidelity: a score whose connection to the claimed capability can be a thin fraction of its face value.
Construct validity has the same rot. Simply reordering the answer choices on MMLU drops some models' accuracy by up to 13 percent. Whatever quantity moves when you shuffle the letters A through D, it is not knowledge of microeconomics.
And then there is saturation, quantum's story again in miniature. Frontier models now cluster between roughly 88 and 94 percent on the original MMLU. A six-point band, much of it within run-to-run and prompt-sensitivity noise, is a ruler whose markings have worn off. The instrument can no longer discriminate between the things it ranks, exactly the way XEB's certifying power collapsed once classical methods could match its scores.
Even the field's response rhymes. Quantum retreated from “supremacy” to carefully hedged “advantage” claims on new tasks. AI is migrating to GPQA Diamond, SWE-bench Verified and Pro, ARC-AGI-2, AIME 2025, Humanity's Last Exam: harder, cleaner, contamination-resistant successors. Both migrations are the same quiet admission. The old instrument had low fidelity to the construct, and everyone with a stake in the numbers eventually knew it.
Sycamore's value to the AI world is that its story is complete. We know how each act ended, which lets us use it as a predictor. Three predictions transfer directly.
First: the headline outlives its own refutation. “Ten thousand years” was repeated for years after IBM cut it to 2.5 days; it still shows up in talks today. “Model X scores 92 on MMLU” will circulate long after the contamination audits that hollowed it out. Budget for this in your own organization: the soundbite that justified a decision will survive the death of the measurement behind it, and someone has to be assigned to notice.
Second: the gap gets erased by a boring baseline, not a rival breakthrough. Nobody out-quantumed Sycamore. GPUs and better linear algebra did it. Likewise, the thing that deflates a benchmark lead is rarely a competing model; it is a cheaper explanation of the same score. Contamination. Retrieval. Answer-order heuristics. A test suite that passes wrong answers. The killer question for any impressive number is not “can anyone beat this?” but “what is the dumbest process that produces this score?” In both fields, the dumb process turned out to be embarrassingly competitive.
Third: a score on a spoofable instrument is a hypothesis, not a result. Once XEB was shown to be classically gameable, every past and future XEB-certified claim inherited an asterisk. AI benchmarks are gameable from several directions at once, from training on the test to fitting the format instead of the task. This does not mean scores are worthless. It means their epistemic status is “promising, pending adversarial verification,” and any culture that treats them as settled results is storing up walkbacks.
Here is the part of the quantum story that deserves more envy than the chip does.
The reason we know Sycamore's claim eroded, in such precise, dated increments, is that the classical simulation community treated the claim as an attack target. IBM within days. Pan, Chen, and Zhang in 2021, with code and fidelity figures. The Sunway supercomputer team. The 2024 A100 result. The spoofing papers. Each counter-result was public, reproducible, quantified, and aimed squarely at the strongest standing claim. Verification was adversarial, continuous, and prestigious: knocking down a supremacy claim was itself a publishable achievement.
AI evaluation mostly lacks this. We have leaderboards, which reward climbing, and we have scattered contamination audits, which arrive late and get a fraction of the attention. What we do not yet have is a standing culture where “I reproduced your benchmark score with a method that lacks the claimed capability” is a celebrated, career-making genre of result. The incentives run the other way: labs grade their own homework, benchmark maintainers depend on adoption, and the audits that do land, like the SWE-bench leakage numbers, spread at a fraction of the speed of the scores they undermine.
The fix is not a better benchmark. Benchmarks saturate and leak; that is their life cycle. The fix is quantum's social technology: institutionalized adversarial verification, where every headline score attracts funded, credited attempts to reproduce it without the claimed capability. Red teams for evals. Bounties for demonstrated contamination. Held-out private test sets rotated like cryptographic keys. And an expectation, written into how results are reported, that “state of the art” is a timestamped claim with a half-life, not a coronation.
If you are a developer or a tech lead consuming benchmark claims, Sycamore hands you a three-question checklist, and none of the questions requires a physics degree.
One: what is the instrument's fidelity to the thing I actually care about? Not the score, the connection between the score and the capability. For a coding model, that means asking how the benchmark's tests were built, whether its solutions leak, and what a passing patch actually proves. If the vendor cannot tell you, the fidelity is unknown, which for planning purposes is the same as low.
Two: what is the boring baseline? Before believing a gap, ask what retrieval, memorization, or format-fitting would score on the same instrument. If nobody has run that baseline, the gap is an estimate from the 10,000-year genre: an upper bound on ignorance, not a measurement of capability.
Three: has anyone been paid to break this? A number that has survived adversarial attempts to reproduce it cheaply is evidence. A number that has never been attacked is a press release. Prefer evals with named critics.
And if you are building internal evals for your own systems, do the one thing neither a vendor's marketing nor the leaderboard economy will ever do for you: hold out a private test set that never touches a training corpus, rotate it, and treat any score within the noise band of your last measurement as a tie. Your eval is an XEB. It certifies nothing by itself. It becomes trustworthy exactly to the degree that someone tried, seriously and recently, to fool it.
Sycamore was real. The chip did what the paper said. What fell was the certainty, and it fell in the only direction certainty ever falls when it stands on a 0.2 percent signal: slowly, then completely, and to opponents nobody considered glamorous. The AI leaderboard nearest you is running the same experiment right now. The only open question is whether you find out its fidelity from your own checklist, or from the walkback.
Your eval is an XEB. It certifies nothing by itself. It becomes trustworthy exactly to the degree that someone tried, seriously and recently, to fool it.
That is the problem the Agent Rating Protocol is built for: a way to rate and rank agents that treats the rating as an adversarial object, one that has to survive attempts to game it rather than assume them away, so a score means the capability and not the exploit. It is one layer of the Agent Trust Stack, the harness for making agent behavior verifiable, rateable, and claimable rather than taken on the leaderboard's word.
Read the Theory of Agent Trust
pip install agent-rating-protocol · npm install agent-rating-protocol
Full trust stack: pip install agent-trust-stack · npm install agent-trust-stack