← Back to blog

The Winner's Curse of Model Bakeoffs

A 1971 oil-auction result, a 2020 Nobel, and the Leaderboard Illusion all say the same thing: the top of your benchmark is the one number guaranteed to be optimistic, and the harder you searched, the more it lies.

Published July 2026 · 11 min read

In 1971, three petroleum engineers, Capen, Clapp, and Campbell, published a paper in the Journal of Petroleum Technology with a strange complaint. Oil companies bidding on offshore oil leases were using the best geological science of the age. Their survey estimates were, on average, about right. And yet the companies kept winning auctions and losing money on the tracts they won, in the engineers' words, earning unexpectedly low returns "year after year."

The paper's diagnosis was not that the geologists were bad. It was stranger than that: the companies were losing money because their estimates were good, unbiased, honest, scattered evenly around the truth, and because they kept winning.

Here is the trap. An oil tract is worth roughly the same to every bidder; nobody knows that value, so each firm bids on its own noisy estimate. Some estimates land high, some low. Who wins the auction? The firm whose geologists were most optimistic. Winning, in other words, is not a random event. Conditional on winning, your estimate wasn't just an estimate, it was the highest estimate in the room, and the highest of several unbiased guesses sits systematically above the truth. The very act of winning is evidence you overpaid. Auction economists later compressed the whole phenomenon into one line: you win, you lose money, and you curse.

Fifty years later, the same trap sits in the middle of machine-learning practice, wearing different clothes. It's called a model bakeoff. You evaluate five models, or fifty configurations, or a thousand hyperparameter trials, on a benchmark, and you deploy the top scorer. That top score is the highest of many noisy estimates of true capability. You have just awarded the lease to your most optimistic geologist, and the disappointment that follows in production has the same mathematical shape as the red ink in the 1971 oil patch.

The maximum of noisy estimates is a liar

Strip the story to its skeleton and it's a fact about order statistics. Take one unbiased measurement of a quantity and your expected error is zero. Take N unbiased measurements and keep only the largest, and your expected error is positive: the max sits above the truth, and the gap widens as N grows. Nothing dishonest happened anywhere in the pipeline. Every individual measurement was fair. The bias was manufactured entirely by the selection.

This has an ugly corollary that most people's intuition gets exactly backwards: the harder you search, the worse your winner's score deceives you. Winning a three-bidder auction is mildly suspicious news about your estimate. Winning a fifty-bidder auction means you were the most over-optimistic of fifty. The experimental economics literature made this concrete: in laboratory common-value auctions, John Kagel and Dan Levin found the curse barely appears with three or four bidders but bites hard with six or seven, the opposite of the comforting belief that more competition disciplines prices. More competitors don't just lower your odds of winning; they degrade what winning means. (Economists still argue over how much money the oil patch really lost to the curse, since field data are muddied by survivorship and the option value of leases, but the laboratory effect is robust and replicable, and the mechanism itself is arithmetic.)

Auction theory's remedy is called bid shading, and it earned its architects a Nobel Prize. Robert Wilson showed in 1969 that a rational bidder in a common-value auction should bid below their own estimate, and shade more as the number of rivals grows, precisely to price in the bad news that winning would convey. Wilson and Paul Milgrom shared the 2020 economics Nobel largely for building auction formats that manage this information problem. The deep move is worth staring at, because it feels almost paradoxical: the optimal bid depends not just on what you know, but on what winning would tell you. You must make decisions conditioned on a hypothetical event ("suppose I turn out to be the winner; what does that imply about my estimate?") and the implication is always: it was too rosy.

Now replace "bidders" with "models in your bakeoff" and re-read the paragraph above.

Your leaderboard is a common-value auction

The mapping is nearly exact. A model's true production capability plays the role of the oil in the ground, a common value nobody observes directly. The benchmark score is the geologist's survey: a noisy, roughly unbiased estimate of that value. Running a bakeoff and shipping the top scorer is holding a sealed-bid auction and awarding the lease to the highest estimate. The winner's headline number is inflated not because anyone cheated, but because you selected it for being the maximum: you selected for the favorable noise, and favorable noise does not persist into production.

This is not a metaphor stretched over a gap. The mechanism has been caught operating at every level of the ML stack, and the paper trail is specific.

Start with the version you can watch in real time. Kaggle competitions split their test data into a public leaderboard, visible throughout the contest, and a private one revealed only at the end. Competitors tune against the public board for weeks, thousands of teams, each effectively running a giant search whose scoring function is a noisy estimate of true generalization. When the private board flips on, the community braces for what it calls the "shake-up": the public-leaderboard rankings reshuffle, and public leaders routinely tumble. It's folklore with a metric, since competitors literally measure a contest's shake-up as the average rank change between boards. The public winner was often just the contest's most successful overfitter: the bidder with the rosiest survey.

Then there's the version hiding in your hyperparameter sweep. In 2019, Jesse Dodge and colleagues published "Show Your Work" (EMNLP), a paper about what the single reported "best run" conceals. Their fix was to report expected best performance as a function of search budget, and the demonstration case should unsettle anyone who has ever declared a bakeoff winner. On their text-classification benchmarks, with a tuning budget under roughly ten trials, plain logistic regression beat a convolutional neural network; give both models a larger budget and the CNN pulls ahead. Which model is "best" flips depending on how hard you searched. A bakeoff's verdict is not a property of the models alone. It's a property of the models plus your search effort, which is exactly what the order statistics predicted, since the expected maximum rises with N.

Scale up to the industry's shared leaderboards and the auction becomes literal. "The Leaderboard Illusion" (Singh et al., 2025, arXiv:2504.20879) documented how Chatbot Arena, the community's de-facto public bakeoff for LLMs, was being played: providers privately test many model variants and disclose only the winner. In the most striking case, the authors report that Meta tested twenty-seven private Llama-4 variants before revealing its pick. Whatever you think of the practice, recognize its statistical identity: best-of-27-report-the-max is a textbook winner's-curse machine. The reported score is the maximum of twenty-seven noisy draws, and everything we know about that maximum says: expect the deployed reality to sit below it. And the 2026 leaderboard state makes the statistics unusually visible. On today's public LLM leaderboards, the top handful of models routinely sit within overlapping confidence intervals (rank gaps of a few Elo points against uncertainty spans many times wider, as current benchmark-methodology reviews note) which means the rank ordering at the very top is, in part, literally noise, and crowning the "winner" is selecting the top draw from overlapping distributions. Saturation makes it worse, not better: with MMLU and HumanEval now clustered above 90% across frontier models, the classic benchmarks no longer separate the field, so whatever gap remains between first and second place is composed almost entirely of the noisy margin that reverts, and studies replaying the fresh-test-set exercise in the LLM era report accuracy drops of up to roughly 13% off the familiar numbers. One frontier release this year drew public scrutiny after independent evaluators flagged statistically unusual score patterns on well-known benchmarks: the "winner trained on the test" suspicion now arrives as a news cycle, not a footnote. By 2026 the field had begun to formalize the repair. One recent paper ("Correcting the Winner's Curse in Adaptive Benchmarking," arXiv:2605.05973) proposes explicit statistical corrections for exactly this reuse-and-select inflation, which tells you the problem has graduated from folklore to methodology.

The curse even runs inside the models. Best-of-N sampling, generate N candidate answers and let a reward model pick the best, is argmax over noisy estimates executed at inference time. Gao, Schulman, and Hilton's "Scaling Laws for Reward Model Overoptimization" (2023) measured what happens as N and optimization pressure grow: the reward model's score of the chosen output keeps climbing while the true quality (judged by the gold-standard evaluator) rises, stalls, and then falls. Past a point, more search against an imperfect scorer buys you more of the scorer's errors, not more value. Auction theory would have predicted the shape of that curve in 1971: the more candidates in the room, the more the winner's estimate is luck.

The general law: post-decision surprise

In 2006, James Smith and Robert Winkler pulled the auction result out of auctions entirely. Their paper "The Optimizer's Curse" (Management Science) proved the general case: whenever a decision-maker evaluates several options and picks the one with the highest estimated value, the chosen option's realized value will systematically fall short of its estimate, even when every single estimate is unbiased. They named the lived experience of this theorem "post-decision surprise," which may be the most quietly devastating phrase in decision science. The disappointment that follows your choices isn't (only) bad luck or bad vibes. It's arithmetic. Choosing the best-looking option means choosing the option whose noise broke upward, and noise reverts.

Once you have the pattern, you see it running through an engineering organization like a load-bearing wall you'd never noticed. The vendor that won your RFP bakeoff won partly by being the most optimistic about itself. The A/B tests you chose to ship were selected for having the largest measured effects, so the measured effects overstate what you'll actually harvest, and the experimentation literature now documents this as its own winner's curse, with launched experiments carrying upwardly biased estimates and too-narrow confidence intervals. The RAG pipeline that re-ranks retrieved chunks and keeps the top one; the agent that scores five candidate plans and executes the best; the eval harness that compares this week's fine-tune against last week's across forty models: every argmax over a noisy score pays the same tax, and pays more the wider it searches.

And here is the cultural kicker, the reason this deserves an essay rather than a footnote: our instincts award extra confidence to exactly the number that deserves less. The winning score gets the press release, the postmortem slide, the promotion narrative. Winning feels like evidence of merit, and it is, partly. But it is also, always, evidence of luck, in a proportion set by how noisy your measurement was and how many candidates you compared. The top of the leaderboard is the one place on the leaderboard where the number is guaranteed to be optimistic.

How to shade a bakeoff

The satisfying thing about a fifty-year-old problem is that the fixes are mature, and the auction fix, the statistics fix, and the ML fix turn out to be the same move performed in three dialects.

Shrink before you argmax. Smith and Winkler's remedy is Bayesian shrinkage: before comparing options, pull every estimate toward a prior, and pull noisier estimates harder. In bakeoff terms, a model evaluated on 200 samples should be discounted toward the pack far more than one evaluated on 20,000, before you crown anything. If a small, noisy eval says the new model is 4 points better, your working belief should be "some fraction of 4, possibly a small one."

Re-estimate the winner on data it didn't win on, and expect the drop. The score that selected the winner can't certify the winner; it's contaminated by the selection. Re-run finalists on a fresh held-out set, a second sealed auction the candidates have never seen, and treat that number as the real one. Budget emotionally for the haircut. When Recht, Roelofs, Schmidt, and Shankar built genuinely new ImageNet and CIFAR-10 test sets by replicating the original collection process (2019), accuracies dropped 11 to 14 points on ImageNet, though, honestly told, their evidence attributes the drop mostly to slightly harder images rather than to a decade of leaderboard gaming, and model rankings largely survived. That's the well-calibrated version of this essay's thesis: the absolute number you selected is reliably inflated; the ordering is corrupted only sometimes, and clean, large, fresh evaluations keep the curse small. The mechanism is a law; its magnitude is an engineering variable you control.

Account for N, out loud. The winner of a thousand-way search is far more inflated than the winner of a three-way comparison, so N belongs in the report. This is Dodge's "show your work" discipline: publish expected-best-versus-budget, not the single lucky run. Practitioners have converged on the same instinct from the deployment side: pick against a small, fixed set of two or three benchmarks rather than everything with a leaderboard (more search surface is more curse), and prefer harder, contamination-resistant holdouts, the GPQA-Diamond and SWE-bench-Verified class, over saturated classics. It's also the honest-disclosure fix for shared leaderboards: say how many variants you tried. A score labeled "best of 27" can be discounted rationally; the same score labeled as a single submission cannot, and the difference between those two labels is exactly the information the winner's curse feeds on.

Protect the test set from your own enthusiasm. Every time a benchmark's items leak into your tuning loop, the benchmark's noise becomes something you can select for, and the curse compounds. Nested cross-validation, separate selection and certification sets, and adaptive-analysis tools like the reusable holdout exist to keep the auction sealed. Rational practice, it's worth saying plainly, already is the shading: a team running honest holdouts and reporting variance has priced the curse in. The curse, like its auction original, is what befalls those who report the naked max.

The petroleum engineers' deepest insight wasn't the mechanism, it was the posture. Capen, Clapp, and Campbell told an industry full of confident winners that winning itself was information, and that the rational response to good news about yourself is a specific, calculated dose of disbelief. Auction theorists turned that posture into equilibrium strategy; decision analysts turned it into shrinkage; careful ML teams turn it into holdout discipline. The posture is the transferable part. The next time your bakeoff crowns a champion (a model, a vendor, a config, a plan scored best-of-five by another model) let the celebration last a full minute. Then remember what the number that won is made of: some skill, and precisely the luck you selected for.

Be a little sadder when you win. It's the winning move.

← Back to blog