Climate modeling and AI compression are one engineering move with two names: replace an expensive process with a cheap function fit to its behavior, and inherit the consequences of what you threw away. One field has been living with the failure modes for fifty years.
In November 2020, a team of climate scientists published a result that should be pinned above the desk of everyone who has ever shipped a compressed model. Brenowitz and colleagues had trained two machine-learning emulators to stand in for expensive atmospheric physics inside a climate model: a neural network and a random forest. On the offline test set, the neural network won. It fit the held-out data better. It was, by the numbers every ML practitioner reports, the better model.
Then they plugged both into the running atmosphere. Coupled into the climate model at 200-kilometer resolution, the random-forest version ran stably. The neural-network version crashed the simulated planet within seven days.
The mechanism, when they traced it, is the part worth memorizing (Brenowitz et al., arXiv:2011.03081). Coupled to the model's wave dynamics, the network's small errors fed back into its own inputs, and the feedback loop marched the system into states unlike anything in the training data. The network then did what networks do on inputs far from their training distribution: something confidently wrong, which fed back again. The offline benchmark had tested the model in the one condition that could not reveal this failure, because a static test set never lets the model's outputs become its next inputs. The benchmark measures the open loop. Deployment is the closed loop. And in the closed loop, the more expressive model was the unstable one.
If you build AI systems, that story should feel less like a curiosity from another field and more like a postcard from your own future. Because the two fields are, in a precise sense, doing the same thing.
A global climate model cannot resolve a cloud. Its grid cells run roughly 100 to 250 kilometers on a side; the convection that actually makes weather happens at the scale of meters, and the droplet physics at microns. Nor can you simply refine the grid: halving the horizontal spacing of a coupled model multiplies compute cost by roughly sixteen, because you get four times the cells and the timestep must shrink too, and a century-long high-resolution run already costs millions of CPU-hours. So climate modelers do something called parameterization: they replace the unresolvable fine-scale physics with a cheap function of the coarse variables the model does track. You do not simulate the cloud. You simulate what the cloud does to the grid cell.
A frontier network cannot be cheaply served. So engineers distill: they train a small student to reproduce the input-output behavior of a large teacher. You do not reproduce the computation. You reproduce what the computation does to the output.
Same epistemic move, made independently, under different names: replace an expensive process with a cheap function fit to its behavior, and inherit the consequences of what you threw away. The most famous production example in AI is DistilBERT, which kept 97% of BERT's language-understanding performance on GLUE with 40% fewer parameters and 60% faster inference (Sanh, Debut, Chaumond and Wolf, 2019). The right way to think about that artifact is not “a small BERT someone designed” but “BERT seen through a small aperture.” Climate modeling has been building small apertures onto expensive physics for fifty years.
An honest boundary before going further: classical parameterizations are not distillation. A scheme like Arakawa-Schubert was derived by hand from physical reasoning, closer to an expert-written heuristic than to a student fit to a teacher. The identity gets tight in two places. One is the machine-learning era of climate modeling, roughly 2018 onward, where networks are literally trained on the outputs of high-resolution physics. The other is older and stranger, and it shows how deep the parallel runs.
In 2001, two papers (Khairoutdinov and Randall in Geophysical Research Letters; Grabowski in the Journal of the Atmospheric Sciences) introduced what the field calls superparameterization: embed a small two-dimensional cloud-resolving model inside each grid column of the coarse global model, and have the coarse model query it for the effects of convection. Look at that architecture with machine-learning eyes. There is an expensive teacher that knows the fine-grained truth. There is a cheap student that runs at scale. The student consults the teacher for the signal it cannot compute itself. The modern ML variant simply swaps the embedded teacher for a neural network trained to mimic it. Two communities built the same scaffolding, named it different things, and did not notice each other for over a decade.
When Hinton, Vinyals and Dean formalized distillation in 2015 (generalizing a 2006 idea from Buciluǎ, Caruana and Niculescu-Mizil), the heart of it was “dark knowledge”: train the student not on hard labels but on the teacher's softened probability distribution, which encodes the relative judgments the teacher makes about non-winning classes. Climate science even has an analog of that. A grid cell's mean temperature and humidity are its hard labels; the full distribution inside the cell (which parcels are saturated, how vertical velocity covaries with buoyancy) is the soft structure that determines whether convection actually fires. Schemes like SHOC (Bogenschutz and Krueger, 2013) and CLUBB explicitly track those subgrid probability distributions. Climate scientists were writing down dark knowledge years before machine learning handed them the word.
So the thesis is not an analogy hunting for a rhyme. It is one engineering move with two names, and the useful consequence is this: climate science has been living with the failure modes of that move for decades, at planetary stakes, and has written down lessons AI compression is only now rediscovering.
Here is the lesson I find most valuable, because at first glance the two literatures flatly contradict each other.
From climate: in 2018, Rasp, Pritchard and Gentine published a landmark result in PNAS. Their neural-network parameterization, trained on a superparameterized model, produced a precipitation distribution that “closely matches that of SPCAM, including the tail,” where conventional hand-written schemes show “too much drizzle and a lack of extremes.” Read that carefully: the compressed model beat the hand-crafted one precisely on extreme events. Compression preserved the tail. (It was also around 20 times faster than the superparameterized physics it replaced, and roughly 8 times faster at the whole-model level once radiation is included. Not the four-orders-of-magnitude speedup that sometimes gets attached to this literature; the honest figures are impressive enough.)
From AI: in 2019, Hooker, Courville, Clark, Dauphin and Frome asked what compressed networks forget, and found that compression's damage is not spread evenly. It concentrates on the underrepresented long tail of the data distribution: the rare, atypical examples. They even gave the casualties a name, Pruning Identified Exemplars. Compression destroyed the tail.
Both results are solid. The resolution is that they concern two different tails, and the distinction is worth more than either finding alone.
Rasp's preserved tail is a tail of the output distribution: extreme rainfall intensities. Those extremes were densely sampled in training, because the teacher simulated plenty of storms. The network had every opportunity to learn them, and it did. Hooker's lost tail is a tail of the input distribution: rare kinds of cases the model seldom saw. And the climate paper confirms this reading inside its own results, because the same network that nailed the rainfall extremes was, in the authors' words, “unable to generalize to much warmer climates.” Pushed about four kelvin beyond its training climate, it broke down. Familiar extremes: preserved, even mastered. Unfamiliar inputs: lost.
Now say the alarming part plainly. The entire purpose of a climate model is to project a climate that has not happened yet. The compressed model was strongest exactly where the teacher was well-sampled and weakest exactly where the model exists to be used: the novel case. The same sentence transfers to AI word for word. A distilled student shines on the distribution it was fed and fails on the inputs nobody thought to include, and no aggregate benchmark number distinguishes the two, because the aggregate is dominated by the well-sampled middle. When someone tells you their compressed model “preserves performance on hard cases,” the question that matters is: which tail? Hard outputs it saw often, or hard inputs it barely saw? Those are different claims, and only one of them survives contact with novelty.
Climate science also ran ahead of AI into a subtler trap: what happens when you optimize a lossy model against a benchmark for years.
In 2013, Suzuki and colleagues showed something uncomfortable in Geophysical Research Letters: a climate model could best match observed temperature trends while adopting parameter values that worst reproduced satellite-observed cloud microphysics. The system got the right answer through compensating errors, one wrong process canceling another. The definitive statement came in Hourdin and colleagues' 2017 survey of model tuning in BAMS, which describes the compounding trap: improving a component of a heavily tuned model can make its measured skill worse, because a genuine fix breaks the web of compensations, and the harder a model has been tuned, the harder it becomes to demonstrate that any real improvement is one.
Sit with that in AI terms. A model tuned hard enough to a metric becomes resistant to genuine improvement. Every real fix looks like a regression, because the errors it removes were load-bearing. The AI community is currently rediscovering pieces of this under the names benchmark contamination and eval overfitting; the climate literature stated the mechanism more precisely, a decade earlier. And it points at the same discipline: skill on the tuned metric is evidence about the tuning, not about the model.
Which leads to the healthiest habit the climate community has and AI mostly lacks. The defining climate quantity, equilibrium climate sensitivity (the long-run warming from doubled CO2), is dominated by exactly these parameterizations, above all clouds. Across the current CMIP6 generation of models, the raw spread runs from 1.8 to 5.6 kelvin (Zelinka et al., 2020), wider than the previous generation's 2.1 to 4.7. Volodin (2021) showed that reasonable changes to cloud parameterization inside a single model could span 1.8 to 4.1 kelvin by itself. The students, in other words, disagree wildly, and more compute has not closed the gap, because the bottleneck was never the grid. Yet the IPCC's assessed range in AR6 is narrower than the raw model spread: likely 2.5 to 4.0. Notice the direction of that move. The assessment did not average the models and call it truth. It brought in observations, paleoclimate evidence, and process understanding, and used them to discount model behaviors it had independent reason to distrust. The models are students; the assessment refuses to grade the students by asking the students.
Distillation pipelines rarely have this. The teacher defines the target, the benchmark is drawn from the teacher's world, and there is often no outside referent at all. Climate can discover its teacher was wrong, because the atmosphere gets a vote. Your compression pipeline usually cannot, and that asymmetry should make you humbler about what “97% of teacher performance” means.
Honesty requires the counterweights. The seven-day crash is a discovered principle, not the state of the art: stable neural parameterizations now run across resolutions (Yuval and O'Gorman, 2020, among others), and NeuralGCM (Kochkov and colleagues, Nature, 2024) tracks climate statistics for decades at 140-kilometer resolution, with realistic tropical-cyclone frequencies emerging on their own, at savings the authors describe as orders of magnitude. The field now has a five-billion-example benchmark, ClimSim, for exactly this hybrid work. And the compression story is not uniformly grim: Rasp's 2018 network conserved column moist static energy to a remarkable degree without ever being told to, a physical invariant emerging from imitation alone. Students sometimes learn structure nobody put in the curriculum. The point is not that compression fails. It is that fifty years of compressing the atmosphere taught one field exactly where the costs hide, and the map transfers.
Four habits, each one a climate lesson wearing AI clothes.
Evaluate the closed loop, not just the open one. The offline benchmark cannot see feedback failures, and agentic deployment (where a model's outputs become its own next inputs through tool calls, memory, and multi-step plans) is exactly the coupled simulation. Before trusting a distilled model in a loop, test it in the loop: long rollouts, self-conditioning, the model marinating in its own outputs. The better offline model can be the one that diverges.
Audit the two tails separately. Ask of any compressed model: how does it do on rare but well-sampled outputs (probably fine), and how does it do on rare, undersampled inputs (this is where it quietly died). Build the second test set deliberately, because no aggregate metric will build it for you.
Ask the tuning question of your own evals. How hard has this system been optimized against this metric, and if someone genuinely improved a component tomorrow, would the metric reward them or punish them for breaking the compensations? If you cannot answer, your benchmark score is measuring your tuning history, not your model.
Keep an outside referent. The IPCC narrows the model range using evidence the models never saw. Find your equivalent: human evals off-distribution, live user outcomes, adversarial probes drawn from outside the teacher's world. A student graded only on the teacher's syllabus will look exactly as good as the syllabus, and no better, forever.
Climate scientists did not choose to compress; the atmosphere forced them, and the forcing made them serious about the costs decades before AI made compression an elective convenience. That is the real gift here. When a field has no choice but to live inside its approximations, it learns precisely how approximations betray you. The parameterization literature is a fifty-year field manual for the thing your team started doing three years ago. Read it before your seven days are up.
A student graded only on the teacher's syllabus will look exactly as good as the syllabus, and no better, forever.
That is the problem the Agent Rating Protocol is built for: a way to rate and rank agents that refuses to grade the students by asking the students. It scores an agent against adversarial probes drawn from outside the teacher's world and on the undersampled input tail where a compressed model quietly dies, so a rating reflects behavior under novelty rather than skill on the well-sampled middle. It is one layer of the Agent Trust Stack, the harness for making agent behavior verifiable, rateable, and claimable rather than taken on the benchmark's word.
Read the Theory of Agent Trust
pip install agent-rating-protocol · npm install agent-rating-protocol
Full trust stack: pip install agent-trust-stack · npm install agent-trust-stack