An optimization is only correct for the cost it was told to minimize. The moment your real cost is different, the famous answer is quietly wrong.
In March 2022, a team at DeepMind published a result that should have been humiliating and was instead just clarifying. They took their own flagship language model, Gopher, a 280-billion-parameter giant, and beat it with a model four times smaller. The new model, Chinchilla, had 70 billion parameters. It won not because it was cleverer in architecture but because it had been fed far more text, 1.4 trillion tokens against Gopher's 300 billion, on the same compute budget. And in beating Gopher, Chinchilla quietly indicted almost every famous model of the preceding two years, including GPT-3, as radically mis-built. Not badly trained. Mis-proportioned, at the level of the recipe.
Here is the strange part, the part that makes this a genuinely good story rather than a correction notice. GPT-3 was not built carelessly. It was built in careful obedience to a scaling law, a real one, published by serious people, that said to do exactly what GPT-3 did. Two years later a second scaling law, equally serious, said the first was wrong and to do nearly the opposite. And then practice, the actual models you use every day, ignored both of them and did a third thing that appears in neither paper. Three laws, each overturning the last, and the one everybody now follows was never written down as a law at all, because it optimizes something the scaling papers never measured. That progression is the whole story, and its ending is a lesson that outlives language models entirely: an optimization is only correct for the cost it was told to minimize, and the moment your real cost is different, the famous answer is quietly wrong.
In January 2020, Jared Kaplan and colleagues at OpenAI published "Scaling Laws for Neural Language Models," and it was a genuinely beautiful piece of empiricism. Across more than seven orders of magnitude, model performance fell on clean power-law lines against model size, dataset size, and compute. The headline for anyone with a compute budget was a prescription: given more compute, grow the model much faster than the data. Their compute-optimal model size scaled as roughly C to the 0.73 power, and the data as only C to the 0.27. The paper's own summary was blunt and consequential: "larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data."
Read that sentence again, because it is the intellectual seed of the entire giant-model era. It says, in effect, that parameters matter more than tokens, that if you have a fixed pile of compute you should spend it on being enormous rather than on reading widely. And the industry listened. GPT-3, later that same year, was 175 billion parameters trained on about 300 billion tokens. That is roughly 1.7 tokens for every parameter. It was a spectacular machine and it was, by the letter of the law it was built on, doing the right thing: very large, trained on relatively modest data. The law said big beats well-read. GPT-3 was the law's proof of concept, and its success made "just make it bigger" the reflexive strategy of an entire field.
Then came Chinchilla. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, and their DeepMind coauthors ran a much larger and more careful sweep, over 400 models, and arrived at a flatly contradictory prescription. Model size and training tokens, they found, should be scaled equally: for every doubling of the model, double the data. Their optimal model size grew as C to the 0.50, and so did the data, the two exponents now matched instead of lopsided. At their optimal point the ratio worked out to about 20 tokens for every parameter, more than ten times the token diet GPT-3 had been given.
And they did not leave it as theory. Chinchilla, 70 billion parameters on 1.4 trillion tokens, uniformly and significantly outperformed Gopher at 280 billion, GPT-3 at 175 billion, Jurassic-1 at 178 billion, and Megatron-Turing NLG at a colossal 530 billion, hitting 67.5 percent on the MMLU benchmark. Their conclusion named the era's mistake directly: "current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant." Run the arithmetic on GPT-3 and it stings. At 175 billion parameters, Chinchilla's 20-to-1 ratio implies GPT-3 wanted roughly 3.5 trillion tokens. It got 300 billion. The defining model of its moment was undertrained by nearly twelvefold, starved of data in faithful service to a law that turned out to point the wrong way.
The tidy version of this story stops here: Kaplan guessed, Chinchilla corrected, science advanced. The tidy version is not quite true, and the truer version is more interesting.
In 2024, Tim Pearce and Jinyeop Song published "Reconciling Kaplan and Chinchilla Scaling Laws," and their finding is the detail this whole saga is usually told without. Much of the famous disagreement, they showed, was not two rival truths about the nature of learning. It was a counting convention. Kaplan's analysis counted non-embedding parameters, leaving out the embedding and positional parameters, and it was performed heavily at small model scale. In a small model, those embedding parameters are a large fraction of the total, so leaving them out distorts the relationship between "size" and compute, and it biases the exponent toward parameters. Count parameters consistently, total rather than non-embedding, and the two laws move toward each other. As the reconciliation paper frames it, consistent counting reaffirms Chinchilla's coefficients while explaining Kaplan's original overestimation.
Sit with that. Two epochal scaling laws, each of which redirected how a multibillion-dollar industry spent its compute, and a large part of what separated them was whether you included the embedding table in your parameter count at small scale. It is the kind of thing that should make anyone humble about a clean power law with a confident exponent. And the humility cuts both ways: a 2024 replication by Besiroglu and colleagues re-examined Chinchilla's own numbers and contested the confidence intervals on one of its three estimation methods, even as the roughly-20-tokens-per-parameter core held up. The lines were real. The measurement conventions underneath them were not neutral, and convention, not nature, produced a chunk of the disagreement that reorganized the field.
So the industry converged on Chinchilla-optimal, 20 tokens per parameter, and lived happily ever after. It did not. Practice blew straight past Chinchilla in the other direction, and it did so on purpose.
Look at Llama 3, released by Meta on April 18, 2024. Both sizes, the 8-billion and the 70-billion, were pretrained on more than 15 trillion tokens. Do the division. The 70B model saw roughly 214 tokens per parameter. The 8B model saw roughly 1,875. Against Chinchilla's compute-optimal 20, that is between ten and nearly a hundred times more data than the law recommends, poured into models deliberately kept small. This was not an accident or an excess of caution. Meta said the quiet part out loud in the release: they observed that both the 8B and 70B models kept improving log-linearly even after being trained on 15 trillion tokens, well past the point Chinchilla would have called them finished. The law said stop; the loss curve was still falling; and because they intended to serve these models rather than shelve them, they kept feeding the small ones long after "optimal." It is now the standard move. Nobody training a model they actually intend to deploy trains it Chinchilla-optimal.
So the arc is complete and it is genuinely three-act. Kaplan, 2020: grow parameters faster than data. Chinchilla, 2022: grow them equally. Practice, 2023 onward: grow the data enormously and hold the parameters down. Each rule overturned the one before it, and the final rule, the one that actually governs how the models in your applications were built, is written in no scaling paper. To see why, you have to notice the one thing all the scaling laws quietly agreed on, and were all quietly wrong about.
Kaplan and Chinchilla disagreed about the exponent. They did not disagree about the objective. Both were answering the identical question: for a fixed amount of training compute, how do I get the lowest possible loss? Both minimize training loss per unit of training compute. That is a completely reasonable question, and it is the wrong one for almost everybody, because a frontier model is not trained and then filed away. It is served. It answers billions of queries, and every one of those queries costs compute in proportion to the model's size.
Once you price the whole life of the model, training plus inference, the optimum moves, and it moves hard. A smaller model trained on far more data can reach the same loss as a larger Chinchilla-optimal model. The two are equivalent in quality. But the smaller one is cheaper on every single inference call, forever, for the entire deployed lifetime of the model. So you deliberately "waste" training compute, over-training a small model well past the point Chinchilla calls optimal, in order to shrink the parameter count you will pay for on billions of future calls. The extra training is a one-time cost. The parameter savings recur on every request until the model is retired. For anything you serve at scale, that trade is overwhelmingly worth it.
This is why Llama 3 looks the way it does, and it is why "compute-optimal" turns out to be a misnomer for what practitioners want. Chinchilla-optimal is training-compute-optimal. It is deployment-suboptimal. The variable that actually decides how real models are proportioned, inference economics, the cost of serving, appears in neither the 2020 law nor the 2022 one, because neither was measuring it. The exponent debate was a debate inside a cost function that most of the industry had already outgrown. The moment the real objective changed from "minimize training cost" to "minimize training plus a lifetime of serving," both famous laws became advice for a problem nobody was solving.
The practical residue is bigger than language models, but the language-model version is worth stating first, because it is immediately usable. If you are choosing or fine-tuning a model to deploy, do not target any published tokens-per-parameter ratio. Pick the smallest model that clears your quality bar, and then feed it far more data than any scaling law recommends, because you will pay for its parameters on every call it ever serves and you pay for its training data exactly once. The over-trained small model is not a compromise. For a served workload it is the actual optimum, and the scaling papers will actively mislead you toward a model that is too big and too cheaply trained.
Now the larger lesson, the one worth keeping after this year's models are obsolete. Every optimization result, every clean law with a confident exponent, is optimal only for the cost function it was handed. Kaplan and Chinchilla were not sloppy; they were rigorous about the wrong objective for most of their audience, and even their disagreement was partly an artifact of how one of them counted. When you inherit a best practice, a benchmark-topping recipe, a "compute-optimal" anything, the first question is not whether the math is right. The math is usually right. The question is which cost it minimized, and whether that is your cost. If you train once and serve never, Chinchilla is your law. If you serve billions of times, it is precisely the wrong target, and the right one was never published because your objective was not the one on the whiteboard. The scaling laws are a case study in a mistake that has nothing to do with scale: optimizing a real, rigorous, beautifully-measured objective, and shipping the confident answer to a question you were not actually asking.
The math is usually right. The question is which cost it minimized, and whether that is your cost.
A benchmark-topping recipe optimized for someone else's objective is the same trap as a confident agent output optimized for a leaderboard instead of your task: correct for a cost that is not yours. The discipline that catches it is the same one that catches a fluent-but-wrong agent, don't inherit a confident number, verify it against the objective you actually have. The agent trust stack is that discipline built as installable pieces: verification against your ground truth rather than a published claim, a provenance record of what an agent actually did so you can check the work and not the confidence, and ratings that price an output by its track record. Before you deploy on an inherited best practice, ask which cost it minimized, and measure your own.
Read the Theory of Agent Trust
pip install agent-trust-stack · npm install agent-trust-stack
Or the provenance record on its own: pip install chain-of-consciousness / npm install chain-of-consciousness.