Did the prediction markets know which lab would top the leaderboard? Scored against the rule that the incumbent stays, a month out they did not. Inside the last week they priced each change of leader within hours of the news.
On March 24, 2025, with a week left in the month, traders on Polymarket gave xAI a 92% chance of owning the top model on the Chatbot Arena leaderboard when March ended. Google was trading at about five cents. The next afternoon, Google's price stood at 34 cents at 16:00 UTC and 87 cents an hour later. That was the day Google announced Gemini 2.5 Pro, which went straight to number one on the leaderboard. Six days later the market resolved to Google. More than $4.2 million had traded on that one question.
It is a vivid story, but one month is an anecdote. Polymarket and Kalshi ran a "best AI model" contract for every month from December 2024 through August 2026: 21 months, all resolved. That is enough history to grade the crowd the way meteorologists grade weather forecasters, against an honest baseline. The question is simple to state: did the markets know which lab would top the leaderboard, or did they find out at the same moment everyone else did?
The record. Polymarket's main series asks the same question each month, as in "Which company has best AI model end of March?":
Kalshi runs the same question monthly under the series KXTOPMODEL, with one contract per model rather than per company. Both venues publish full price histories through public endpoints. They agree on the winning company in every month where Kalshi listed the winner; in July 2026 none of Kalshi's 11 listed models won, while Polymarket resolved to Anthropic. The winners were:
The forecast. For each event we took every contract's last traded price at four fixed points, 30, 14, 7 and 1 days before resolution, and normalized the prices to sum to one. That gives the crowd's probability for each company at each horizon.
The score is the Brier score, the meteorologist's measure. Glenn Brier proposed it in 1950 for rain forecasts, and we told that story in "Your Eval Leaderboard Breeds Confident Liars". In the multi-company form used here, 0 is a perfect forecast and 2 is a forecaster who was certain and wrong.
The baseline is the meteorologist's oldest one, persistence: tomorrow will look like today. Here that means "the incumbent stays." On the first of each month it names last month's winner with probability one. It scores 0 when the leader holds and 2 when the leader falls. A forecaster that cannot beat it is adding nothing that a calendar could not.
At 30 days before resolution, on the 20 months that had a previous winner, the scores came out like this:
For practical purposes the crowd a month out was the incumbent rule with some hedging around it.
Then the market pulls away. Its Brier score falls to 0.185 at 14 days, 0.109 at 7 days and 0.001 at 1 day, while the rule stays flat at 0.300. By the last day the favourite won 21 times out of 21.
Two cautions apply:
The crowd was rarely confident a month out, and the incumbent rule was confident and mostly right.
One more pattern shows where the attention went. In several months the biggest contract by trading volume was a long shot that did not win:
Volume counts both sides of every trade, so this is churn on cheap contracts, not a crowd backing those labs. It does mean the busiest corner of the market was not where the answer was.
The interesting months are the three in which the leader fell. In each, the market's 30-day price on the eventual winner was low, and the move came after the news, not before it.
February 2025, xAI.
March 2025, Google.
February 2026, Anthropic.
There was a fourth month in which the favourite lost, and it runs the other way. In August 2025 the market's favourite a month out was OpenAI at 51%, with Google at 46%. Google held the top spot, and by two weeks out the market had Google at 93%. The crowd expected a change that did not come.
The pattern across all four is consistent. The market did not see a change of leader coming in any of them. It saw each change soon after it became public: within the hour of LMArena's Grok 3 post, within the day Gemini 2.5 Pro was announced, and within two days of Claude Opus 4.6's release. That is fast, and it is a kind of efficiency. But it is efficiency about information already out in the open.
Friedrich Hayek's famous case for prices, in "The Use of Knowledge in Society" (1945), rests on a particular kind of knowledge, "the knowledge of the particular circumstances of time and place." It is scattered across many people, and no central planner could gather it. "We must look at the price system as such a mechanism for communicating information," he wrote. Prediction markets are built on that argument: many traders, each with a fragment, and a price that adds the fragments up.
The question these contracts ask does not have that kind of knowledge. What decides next month's number one sits with a handful of labs, and with the leaderboard itself.
"The Leaderboard Illusion", a 2025 study of Chatbot Arena by Shivalika Singh and colleagues, documents how lopsided that is:
Labs test anonymously before they launch. Grok 3 was on the arena as "chocolate" before it had a public name, and a lab running its own model under a codename can see how that model is doing. A trader sees a codename.
Sanford Grossman and Joseph Stiglitz made the economic version of this point in 1980. Informed trading needs a reward, so prices can only carry information that someone can acquire and profit from. Here the information that would move next month's price is a lab's private arena results and its release date. No outside trader can buy that. So the price waits for the news, then moves fast. The prices behave like traders reading the leaderboard rather than the labs. I found no report of insider trading on these contracts, and nothing here implies any.
Finance has a plain rule for this: a derivative is only as good as the index it settles on. A "best AI model" contract is a derivative of the leaderboard, so whatever the leaderboard measures, re-labels or changes, the contract inherits.
Over these 21 months, the thing being forecast changed under the traders' feet:
None of this is wrong. Every change was published in the rules. But a forecaster grading these markets is grading a moving target, and that is the eval-integrity problem in miniature: a market on a leaderboard can be no more stable than the leaderboard's own definition of "best."
The churn is sharper at the level Kalshi trades, one contract per model. The same model held number one in consecutive months only 10 times out of 19, because version churn turns the top slot over even when the company does not:
At this level the incumbent rule is weak, and the market beat it clearly inside a week. At 7 days the market's Brier score was 0.695 better (95% interval −1.127 to −0.272), and at 1 day 0.908 better. Both intervals exclude zero. The market's edge there is knowing which build will ship and be listed. By the last week of a month, that is usually public too.
So go back to that 92% for xAI on March 24, 2025. It was not a failed forecast. It was a price on persistence: xAI held the top spot, and nothing public said otherwise. The information that ended it was inside Google until the day it was announced. Then the market priced it within hours.
The lesson applies well beyond betting markets. When someone hands you a probability about a leaderboard, ask two questions before you lean on it:
And since the top slot changes by build far more often than by company, choose a model on your own tasks, not on the name at the top of the list.
Sources
The Agent Rating Protocol records ratings of agents and models as signed, dated entries tied to the work they rate, so a reputation can be checked against its track record instead of a leaderboard snapshot.
pip install agent-rating-protocol npm install agent-rating-protocol
Pair it with provenance: Hosted Chain of Consciousness.