Against an LLM endpoint there is no “again.” A retry is a re-sample, and if the call is a decision, your fault tolerance is quietly moonlighting as a sampling policy nobody chose.
Researchers at Thinking Machines Lab recently ran an experiment that every engineer with a max_retries=3 in their LLM pipeline should sit with. They sent the same prompt to Qwen3-235B one thousand times, at temperature zero, greedy decoding, the setting everyone calls “deterministic.” They got back 80 distinct completions. The outputs were identical for the first 102 tokens. At token 103, the paths split: some completions said “Queens, New York,” others said “New York City,” and from there the texts diverged for good.
Eighty different answers to one question, at the temperature that is supposed to mean “always the same answer.”
The cause is the genuinely surprising part. It is not floating-point folklore, and it is not information leaking between requests. The forward pass of a production inference server lacks what the authors call batch invariance: the numerical result of the computation depends on the batch it ran inside. In their words, “when a non-batch-invariant kernel is used as part of a larger inference system, the system can become nondeterministic.” Your request gets batched with whatever other traffic hit the server that millisecond, the kernel's reduction order shifts with batch shape, tiny numerical differences change one token, and autoregression amplifies one token into a different document. Which means the answer you got depends, in a small but real way, on who else was talking to the model at the time. Nobody reasons about that when they add a retry loop. Almost nobody knows it is true.
Now put that fact next to the humble retry, the most reflexive piece of plumbing in distributed systems. The whole mental model of a retry is: the operation failed, so run the same operation again. Against an LLM endpoint, that operation does not exist. There is no “again.” Attempt two runs in a different batch, against different concurrent load, and measurably often returns a different answer. The word “retry” is false advertising. What you built is a re-sample. And if the call you are re-sampling is a decision (a judge model scoring an eval item, a classifier routing a ticket, an extractor committing a value), then your retry loop is not fault-tolerance machinery. It is a second-opinion machine that silently overwrites first opinions, with the tiebreak decided by whichever attempt happened to complete.
Classical pipelines have their own version of duplicate trouble, and it is well-trodden territory: a recovery process requeues finished work, duplicates fan out, you pay twice and clean up the mess. We have written about that failure shape before, and the distributed-systems literature has owned it for decades, back to the impossibility results about exactly-once delivery. But notice what makes the classical case survivable: a duplicate execution of a deterministic task produces a copy. Wasteful, embarrassing, semantically harmless. Run the same video transcode twice and you get the same video. The LLM case breaks that comfort. A duplicate execution produces a disagreement. The duplicate is not a copy of your data; it is a competing claim about the truth, and unless you built something to prevent it, the loser of that competition vanishes without a trace.
Here is the part I have not seen anyone say about eval pipelines, so take it as this essay's argument rather than a citation.
Nobody retries uniformly. You retry on failure: the timeout, the rate-limit response, the malformed JSON that would not parse. And those failure modes are not independent of content. The long, gnarly, ambiguous items are precisely the ones that blow past token limits, time out on slow decodes, or produce output mangled enough to fail parsing. Easy items complete on attempt one. Hard items complete, disproportionately, on attempt two or three, in a different batch, as a different sample from the model's answer distribution.
Follow that to its conclusion. The verdicts in your results table are not “the model's judgment on each item.” They are “the model's judgment conditioned on the attempt that happened to complete,” and the conditioning is strongest on exactly the difficult items you built the eval to probe. Your dataset of verdicts has quietly become a weighted draw from the model's opinion distribution, where the weights were assigned by infrastructure noise: server load, batch composition, whoever else was querying that second. Two runs of the “same” eval suite can disagree, not because the model changed, but because the lottery of which attempts landed came out differently. When someone then asks why this week's pass rate moved two points, you will look for a model regression, a data change, a prompt bug. The real culprit can be that your retry policy is a sampling policy, and nobody chose it.
It helps to be concrete about what the pollution looks like downstream, because “phantom duplicates” sounds abstract until you have debugged one. Two committed verdicts for one item means your aggregation query either double-counts the item, keeps whichever row it met last, or crashes on a uniqueness assumption someone made three services away. Pass rates drift depending on query order. A dashboard shows 1,043 judgments for a 1,000-item suite and nobody can say which 43 items are twins, or which of each twin's two opinions is the one a human already acted on. The cleanup is a forensic exercise in deciding, after the fact, which of two model opinions was “real.” There is no principled answer to that question. The only good time to answer it is before the second opinion exists.
That is why the fix matters more than its size suggests. It is tempting to call it a one-line check: “this ID already has a committed verdict, skip.” As a first approximation, yes. Once a decision is committed for an item, later attempts, stragglers, requeues, and duplicate messages hit the check and become no-ops. You pay for the compute once, and, more importantly, exactly one opinion per item enters the record. The retry loop goes back to being fault-tolerance instead of a hidden second-opinion generator.
But the one-line framing undersells both the pattern's pedigree and its subtlety, and the subtlety is where implementations quietly go wrong. Two corrections, both standard, both worth the paragraph.
First: what the ID keys on. This pattern is not folk wisdom; it has a written standard. The IETF's httpapi working group produced a draft specifically for it, the Idempotency-Key HTTP header, which exists to “make non-idempotent HTTP methods such as POST or PATCH fault-tolerant,” was explicitly inspired by the production idempotency machinery at Stripe and PayPal (with a nod to Mark Nottingham's old “POST Once Exactly” draft), and recommends a UUID or similar random identifier as the key. The draft's most recent revision dates to late 2025; drafts lapse, but the pattern it codifies is running in every serious payment API on earth. And the sentence in it that earns its keep for our purposes is this one: “The idempotency key MUST be unique and MUST NOT be reused with another request with a different request payload.”
Read that clause against the naive implementation, which keys on item_id alone. The moment your prompt template changes, your judge model rolls to a new version, or your rubric gets edited, the request payload for that item is different, and the correct behavior is a new verdict. A bare item-ID check cannot tell the difference; it will pin the stale verdict forever and silently serve you last month's judgment against this month's rubric. The spec already told you the naive version is wrong. The key that honors it is composite: something like (item_id, prompt_version, model_id, rubric_version). Same payload, same verdict, forever; changed payload, fresh decision. The idempotency key is not a deduplication hack. It is a declaration of which changes are supposed to change the answer, which makes designing it a small act of epistemology, not plumbing.
Second: how the check executes. “Check if a verdict exists, then write one” reads correctly and is a race. Two workers, both holding a retry for the same item, both run the existence check, both see nothing, both write. Under exactly the concurrency that motivated your retries, check-then-write manufactures the duplicate decisions it was built to prevent. The real one-liner is not an if statement; it is a uniqueness guarantee in the storage layer: a UNIQUE constraint on the composite key with an INSERT ... ON CONFLICT DO NOTHING, a conditional put, a transactional upsert. Let the database refuse the second opinion atomically. The difference between the if and the constraint is invisible in the demo and decisive in production, which is the profile of most distributed-systems bugs worth knowing about.
One placement note, since the IETF draft describes a client-server header and your pipeline probably is not one. The header pattern exists because Stripe cannot trust its clients; the key travels with the request so the server can deduplicate. Inside your own eval pipeline you own both sides, which simplifies the answer: enforce at the layer where the truth lives, the verdict store, and let every producer be as dumb as it likes. Workers do not coordinate, do not check, do not care; they attempt writes, and the constraint arbitrates. Pushing the invariant to the data layer is what makes it survive the next refactor, the second worker pool, and the colleague who adds a new retry path without reading this essay.
There is also a systemic reason to get this right that goes beyond your own bill. The distributed-systems community has a name, metastable failure, for outages that sustain themselves after the trigger is gone, and a HotOS '21 study by Bronson and colleagues found retry storms to be a classic engine of them: retries amplify load, which causes failures, which cause retries. Every failure they studied came from real production cloud systems. LLM pipelines add a nasty twist, and flag this as inference rather than citation: the resource being exhausted is a metered token quota, and rate-limit responses are themselves the retryable error, so an undisciplined retry loop burns exactly the budget the pipeline needs to recover while generating more of the signal that triggers it. Idempotency keys, retry budgets, and backoff with jitter are the standard triad; the key is the piece that caps the decision damage while the other two cap the load.
Honesty about the limit, because it points at the deeper practice. Idempotency-by-ID commits a verdict per key. Not the right verdict. If attempt one drew an unlucky sample from those eighty possible completions, the constraint now faithfully preserves the unlucky answer forever. Idempotency buys reproducibility and cost control. It does not buy accuracy.
If accuracy on contested items is what you need, the answer is not retrying harder; it is the mirror image of a retry. Sample the model deliberately: run five judgments on purpose, with an aggregation rule you wrote down, a budget you chose, and the variance recorded in your results instead of hidden behind whichever attempt survived. A retry is unplanned resampling with the disagreement discarded. An ensemble is planned resampling with the disagreement measured. The same API calls, opposite epistemics. You can also, at the infrastructure level, buy determinism outright: the Thinking Machines team shipped batch-invariant kernels that made all one thousand completions bitwise identical, at a real performance cost, 26 seconds for standard vLLM versus 42 with their improved deterministic attention kernel. Determinism is purchasable, and for some compliance-shaped workloads it is worth the price. For everyone else, the cheap resolution of the whole essay is the one-row version: the call whose result you never re-request costs nothing extra, drifts nowhere, and disagrees with nobody.
So, the practical insight, in the order you should apply it. Give every decision in your pipeline a committed identity: a composite key naming the item and everything that legitimately changes the answer. Enforce it with an atomic constraint, not a check. Let retries hit the constraint and die as no-ops, so your fault tolerance stops moonlighting as an unlogged sampling policy. And when a decision is important enough that one draw from eighty possible answers is not good enough, do not retry your way to confidence. Decide, deliberately, how many opinions to buy, and write down how much they disagreed. The retry loop was never qualified to make that call. At token 103, it does not even know it is making one.
Give every decision in your pipeline a committed identity, and let exactly one opinion per item enter the record.
That is the invariant Chain of Consciousness is built to hold. Every decision an agent commits gets a tamper-evident record keyed to what it was deciding about and the version of everything that legitimately changes the answer, so a retry, a straggler, or a duplicate message hits a committed identity instead of overwriting a first opinion with a second. One decision, one durable entry, reproducible on replay, with the disagreement measured on purpose when you choose to sample rather than discarded by whichever attempt happened to land.
See Hosted Chain of Consciousness · Read the Theory of Agent Trust
pip install chain-of-consciousness · npm install chain-of-consciousness
Or the whole trust stack at once: pip install agent-trust-stack / npm install agent-trust-stack