← Back to blog

Your Agent Eval Is One-Factor-at-a-Time, and Fisher Proved That's Blind

Change one thing, hold the rest fixed "for fairness." That is the exact doctrine R.A. Fisher demolished a hundred years ago, and its blind spot is the interactions that run your agent.

Published July 2026 · 11 min read

One afternoon in the 1920s, at an agricultural research station north of London, a scientist named Muriel Bristol refused a cup of tea. Her colleague had poured the milk in after the tea, and Bristol, an algae biologist at Rothamsted Experimental Station, insisted she could taste the difference. Milk first was one drink; tea first was another. The men at the table laughed. One of them, a young statistician with thick glasses named Ronald Fisher, did something better than laugh. He designed a test.

Eight cups. Four poured milk-first, four tea-first, presented in randomized order. Bristol's job was to sort them into the two groups of four. Fisher had done the arithmetic before the kettle boiled: there are exactly 70 ways to choose four cups out of eight, so if she were guessing, her chance of a perfect sort was 1 in 70. As the story was passed down by witnesses, she got all eight right.

The tea is not the point. The point is that the meaning of her answer was manufactured in advance, by the design. Without the randomization, without the balanced four-and-four, without the pre-computed 1-in-70, her perfect sort would have been an anecdote. With them, it was evidence. Fisher's whole career is in that move: the analysis happens before the data. The design of an experiment decides what the experiment can ever tell you.

Which brings me to your agent eval, because there is a good chance it is built on the exact methodology Fisher spent the 1920s demolishing. The failure mode he identified is not a subtle one.

The doctrine Fisher tore down

Here is how careful scientists were taught to work, for generations: change one thing at a time. Hold everything else fixed. Vary the fertilizer while keeping the seed, the plot, and the season constant; then, in a separate experiment, vary the seed. One question per experiment. Anything else was considered sloppy. How would you know which factor caused the change?

In 1926, one hundred years ago this year, Fisher published a paper called "The Arrangement of Field Experiments" in the Journal of the Ministry of Agriculture, and went after this doctrine by name:

"No aphorism is more frequently repeated in connection with field trials, than that we must ask Nature few questions, or, ideally, one question, at a time. The writer is convinced that this view is wholly mistaken. Nature, he suggests, will best respond to a logical and carefully thought out questionnaire; indeed, if we ask her a single question, she will often refuse to answer until some other topic has been discussed."

He then spent a decade building the alternative, formalized in his 1935 book The Design of Experiments. Instead of varying one factor at a time, a protocol now known in the trade as OFAT, you vary all the factors together, in a deliberate pattern that covers every combination. He called the factors "factors" (the word is his) and the method "factorial design."

The statistics community has regarded this as settled for most of a century. Every field that runs physical experiments, from agronomy and chemical engineering to pharmaceutical manufacturing and semiconductor fabrication, treats factorial design as the baseline of competence. Fisher's argument for it had two prongs, one about efficiency and one about blindness. The blindness prong is the one that should worry you.

The ritual we actually run

Picture the eval process at a typical team shipping an LLM agent. It goes something like this.

The new model drops. You swap it into the agent, holding the system prompt, the tool configuration, and the scaffold exactly as they were, "so it's a fair comparison." You run the benchmark suite. The new model scores worse. Someone writes in Slack that the new model "isn't actually better for agentic work," and the team moves on.

Or: the model stays fixed, and someone spends a week on the prompt. Terse version, structured version, version with the XML tags. Best one wins, ships, becomes the new frozen baseline. Later, someone adds a browser tool and reruns the suite, with the model and the now-canonical prompt held fixed, and the score goes up three points, so the tool stays.

Each of those steps is disciplined, controlled, and defensible on its own. Change one thing, measure, conclude. It is also, precisely, one-factor-at-a-time experimentation. The standard guidance in the agent-evaluation literature says it out loud: re-run your offline evals whenever you change "prompts, models, or tool configurations," the changed variable, tested in isolation, everything else pinned. Fisher's target, a century later, wearing a lanyard.

What's wrong with it? Exactly what Fisher said was wrong with it in 1926: it cannot see interactions. And in this field, the interactions are not a correction term. They are the main event.

The cell you never visited

An interaction, in Fisher's sense, is when the effect of one factor depends on the level of another. Not "prompt matters and model matters," but "which prompt is best depends on which model you're running."

Make it concrete with the smallest possible example: two models (call them A and B) and two prompts (terse and structured). That's a 2×2 grid, four cells. The OFAT ritual visits three of them. You start at (A, terse), your baseline. You test the new model: (B, terse). Worse. You separately test the new prompt on the incumbent: (A, structured). A little better. Conclusions: model B is worse, structured prompting helps a bit. Ship (A, structured).

The fourth cell, (B, structured), was never run. If model B happens to be dramatically better with the structured prompt, the entire conclusion is wrong, and nothing in your data can warn you. You didn't measure a property of model B. You measured a property of model B under the prompt you froze for fairness, a slice through the grid, reported as if it were the grid.

Now scale that up to a real agent, where the factor list is model × prompt × tool set × memory strategy × planning loop × temperature. That's a six-dimensional hypercube, and OFAT walks along its axes, one edge at a time, radiating from whatever baseline history happened to hand you. The interior of the cube, where every deployed configuration actually lives, is terra incognita. You will ship a point in the interior. You will have tested points on the edges.

If interactions were small, this would be a forgivable approximation. In agricultural fields they're often moderate. In LLM systems they are enormous, and we have the receipts:

• Sclar and colleagues showed in 2023 that formatting alone, the separators, the casing, the spurious details of prompt layout that any reasonable person would call equivalent, swings benchmark accuracy by up to 76 points on LLaMA-2-13B in few-shot settings. Not 76 basis points. Seventy-six accuracy points, from changes a style guide wouldn't even flag.
• A 2025 benchmark study of prompt sensitivity found the damning half of the result: the best-performing formats do not transfer between models. The prompt that flatters Model A sits near the bottom of the distribution for Model B. That non-transferability is not an inconvenience. It is a prompt×model interaction, measured at scale. It means "which model is better" literally has no answer independent of the prompt, and vice versa.
• A 2025 paper needed only its title to make the point: "A Single Character can Make or Break Your LLM Evals."

Put those together and the OFAT eval isn't just imprecise. It is answering a different question than the one you asked. When you froze the prompt "for fairness" and swapped models, the ranking you got was conditional on that frozen prompt, and the research says the ranking flips under other reasonable prompts. The factor you held fixed is driving the result. This is Fisher's line made literal: ask Nature a single question, and she will refuse to answer until some other topic has been discussed. The model question cannot be answered until the prompt topic has been discussed, jointly, in the same experiment.

The part everyone gets backwards

Here is where most people's intuition betrays them. Factorial design sounds like the expensive, gold-plated option: "we can't afford to run the full grid." Fisher's second prong, the efficiency argument, says the opposite, and it's the most counterintuitive result in the whole story.

In a factorial design, every run does multiple jobs. Take the 2×2×2 case: eight runs covering three factors. When you estimate the model effect, you compare four runs against four runs. When you estimate the prompt effect, you compare a different split of the same eight runs, four against four. Same for the tool effect. Every run participates in every comparison, and statisticians call it hidden replication. To get the same four-versus-four precision on three factors with OFAT, you'd run a baseline plus dedicated variations: more runs, and at the end you'd hold three main effects and zero information about interactions. The factorial got you all three main effects at equal precision, plus every two-way and three-way interaction, out of the same budget. The interactions are free. You were already paying for them and throwing them away.

And when the factor list gets long, you don't need the full grid. In 1946, Robin Plackett and Peter Burman worked out screening designs that estimate the main effects of k factors in roughly k+1 runs: seven factors in eight runs, eleven in twelve. Fractional factorials let you buy the interactions you care about and fold away the ones you don't. This is a mature, century-deep toolbox designed for exactly the situation agent teams are in: many factors, expensive runs, and a boss who wants the answer this week. The irony of the current moment is that eval compute gets burned on serial OFAT ladders, tune the prompt, then swap the model, then re-tune the prompt because the swap broke it, which is the factorial's run count paid in installments, for a fraction of the information, with the interactions billed as "weird regressions" discovered in production.

What a Fisher-grade agent eval looks like

None of this requires new tooling. It requires arranging runs you were going to do anyway into a shape that can answer questions. Concretely:

Pick the factors you actually argue about. If your team has debated it in a design review, it's a factor: model, prompt family, tool set, planning strategy, context policy. Take the top three to five, two levels each. Current versus candidate is enough.

Run the grid, not the ladder. A 2⁴ design is sixteen configurations; with three seeds each, forty-eight runs, often fewer than the meandering OFAT sequence a team burns through in a launch week, and every run lands in a designed cell instead of a Slack thread. Sixteen too many? Run the half-fraction: eight configurations, all four main effects intact.

Replicate and randomize, because agents are stochastic and judges drift. Multiple seeds per cell, variance reported next to the mean. A "best" configuration inside another's error bars is a tie, and pretending otherwise is how eval-chasing starts. Randomize task order within runs; keep the judge version fixed within any comparison you'll cite, and treat judge upgrades the way Fisher treated field-plot soil gradients, a nuisance variable to block, never to ignore.

Read the interaction table before the leaderboard. The deliverable isn't "Model B won." It's "Model B wins under the structured prompt with the browser tool, loses everywhere else, and the prompt effect is three times the model effect." That conditional sentence is the actual deployment decision, and only a factorial can produce it.

And when someone tells you the new model "tested worse," you now have the precise, polite, devastating question: under whose prompt?

A hundred years ago this year, Fisher wrote that Nature refuses to answer single questions. Muriel Bristol sorted all eight cups, and it meant something, not because her palate was sharp, but because a statistician had arranged the cups so that "all eight" could only mean one thing in seventy. Your benchmark is the cups. The design decides what it can ever tell you, and it decides before a single token is spent.

← Back to blog