The Fleet Cookbook — Foreword: Operational Failures as Recipes

Heat egg yolks to about seventy degrees Celsius and you have a sauce. Heat them to seventy-six and you have scrambled eggs. The difference is a number you cannot taste, a transition you cannot reverse, and the line between them is not in the recipe — it is in your wrist. This is carbonara. It is also, in a strict structural sense, every distributed system you have ever shipped.

On July 19, 2024, CrowdStrike pushed a routine update to its Falcon Sensor security software. The new IPC Template Type defined twenty-one input parameter fields. The integration code that loaded Channel File 291 supplied only twenty values to match against them. That gap — one missing field, unchecked because of a bug in CrowdStrike’s Content Validator — caused the sensor to perform an out-of-bounds memory read inside the Windows kernel when it reached for the twenty-first input. Roughly 8.5 million Windows devices crashed simultaneously. That is Microsoft’s estimate, less than one percent of all Windows devices in the world, which sounds small until you remember that one percent included airlines, hospitals, payment terminals, broadcasters, emergency services, and the boarding-pass infrastructure of every major airport in the United States. Parametrix later estimated $5.4 billion in losses to U.S. Fortune 500 companies alone, and that figure does not include the global cascade.

One field. Twenty-one instead of twenty.

This is the same shape as carbonara. The input space is continuous — temperatures, byte arrays, memory addresses, configuration flags. The output space is binary — sauce or scramble, kernel running or kernel panic. There is a threshold somewhere in the continuous space, narrow enough that you cannot see it from the outside, and crossing it costs you everything that depended on the system staying on the correct side of the line.

This essay is the foreword to a cookbook. The cookbook contains fifty real recipes — each named after an operational disaster, each headnoted with the failure mode that inspired it, each technique mirroring the structure of what went wrong with enough fidelity that if you understand why the soufflé fell, you understand why the deployment failed.

Three Properties Most Domains Don’t Share at Once

Cooking and production systems both happen to live in a corner of the world that almost nothing else lives in. Three properties have to be present simultaneously, and most domains have at most two.

The first is irreversibility under time pressure. You cannot un-burn a roux. You cannot un-coagulate a yolk. The Maillard reaction — the browning that produces bread crust, grilled meat, and roasted coffee — is a one-way chemical transformation; the lysine residues that participate in it are permanently alkylated, a finding reaffirmed in the 2025 PubMed Central review of the reaction’s mechanism. Caramelization is technically a form of pyrolysis: sucrose molecules decompose, and once decomposed, they are not sucrose anymore. They are different molecules. There is no backwards.

In production systems, the irreversibility is procedural rather than chemical, but the wall is the same height. The Knight Capital trading firm lost on the order of $440–$460 million in roughly forty-five minutes on August 1, 2012, when a deployment to seven of eight servers reactivated a dormant routine called Power Peg on the eighth. Power Peg had been retired in 2005 and left in the codebase like a knife forgotten in a drawer; a flag repurposed for new functionality reached over and pulled it. By the time anyone understood what was happening, that server had executed about four million unintended trades across 154 stocks, totaling roughly 397 million shares — the SEC’s enforcement record (Press Release 2013-222) reconstructs the timeline. Those trades were binding market transactions. The firm needed a $400 million emergency capital injection within days, lost more than seventy percent of its market value, and stopped existing as an independent company shortly after. You cannot un-execute four million stock trades any more than you can un-burn a roux.

The second is nonlinear response to small input changes. The egg-yolk coagulation threshold sits between sixty-five and seventy degrees Celsius depending on pH, salt content, and the proteins around it. The carbonara window is somewhere around five degrees wide, give or take, and that range comes from cooking the dish rather than from a peer-reviewed paper — call it directional. What is firm is that the input is continuous and the output is categorical. Use yolks alone instead of whole eggs and the threshold rises a few degrees, widening the margin. Cooks who can taste the difference between “almost there” and “done” have moved that threshold around inside their bodies. Cooks who cannot, scramble.

In software, the One-Bit Catastrophe is the same shape. Knight Capital’s $440 million was the difference between a flag set on one server and the same flag set on eight. CrowdStrike’s $5.4 billion was the difference between twenty fields and twenty-one. The 2017 WannaCry worm spread through hundreds of thousands of machines on the back of a single unpatched Windows vulnerability that Microsoft had already issued a fix for. In each case, the input change is something you could write on a sticky note. The output is a number with a “B” after it.

The third is expertise that cannot be fully specified in rules. This is Michael Polanyi’s territory. In The Tacit Dimension (1966), Polanyi made one of the more annoyingly true observations in twentieth-century philosophy: “we can know more than we can tell.” A doctor recognizes a face. A driver feels the road. A baker knows the dough is ready. None of them can give you a procedure that, if followed by someone else, produces the same result. Polanyi’s own example was driving: “the skill of a driver cannot be replaced by a thorough schooling in the theory of the motorcar.” The actual skill lives in the body, in the trained perception, in the calibration that took years to acquire and cannot be transferred through a text channel.

A 2025 paper in the Journal of Nutrition Education and Behavior applied Polanyi’s framework directly to cooking and made the same point clinically: the recipe captures roughly an ingredient list and a procedure; the chef captures everything that happens between the steps. The chef does not look at a timer. The chef looks at the pan and knows. Cognitive science estimates — and these are estimates, not measurements; treat them directionally — suggest a large fraction of an expert’s working knowledge is tacit, which is why expert practitioners are notoriously bad teachers and why apprenticeship persists as a training model in domains where formal instruction has had centuries to displace it.

The operational equivalent of “fold gently until just combined” is “proceed with caution.” Both instructions are, in literal terms, useless. They tell you nothing about what to do. They are placeholders for a competence the writer cannot articulate and assumes you already have. Runbooks are full of this language because the alternative — actually specifying the judgment — is not possible. A senior site-reliability engineer running through a recovery checklist makes thirty micro-decisions that don’t appear on the checklist. A first-week junior running through the same checklist sees thirty steps. The checklist is identical. The outcomes are not.

These three properties — irreversibility, nonlinearity, and tacit expertise — show up together in cooking, in production operations, in surgery, in piloting, in nuclear plant operation, and in a small number of other places. Most domains have one or two. Software development, which most people reach for when they want a comparison to operations, has none of them: development is iterative, the cost of mistakes is bounded, and the keyboarding-level skill is mostly explicit. Production operations are the irreversible cousin of development. The cookbook is about production.

Why the Mapping Is Not Just a Metaphor

Charles Perrow, a sociologist at Yale, published Normal Accidents in 1984. He had spent several years studying Three Mile Island and a number of less famous industrial disasters, and he came to a conclusion the engineering profession was not entirely happy with: in systems with high interactive complexity (where many components can interact in unexpected ways) and tight coupling (where there is little buffer or slack between components), accidents are not exceptions — they are structural features. Perrow called them “normal accidents” because under those conditions, accidents are normal. He meant the word in the sense statisticians mean it: not unusual.

A modern restaurant kitchen and a modern distributed system both sit in the high-complexity, tight-coupling quadrant of Perrow’s matrix. A restaurant on Saturday night has dozens of tickets simultaneously in flight, multiple cooks sharing burners, sauces that need to come out of three pans within a fifteen-second window to plate together, and no slack — if the steak is two minutes late, the asparagus is overcooked. A microservices deployment across three regions has roughly the same shape: many components, dense interactions, no slack. Cooking and systems operations are not similar because they “feel” similar. They are similar because they share a position on Perrow’s coupling-complexity matrix, and Perrow’s framework is about what happens at that position. The shared failure taxonomy is downstream of the shared structural position.

The food industry already knows this, which is the most surprising thing in any of the research that went into this cookbook. HACCP — the Hazard Analysis Critical Control Points framework that governs commercial food safety almost everywhere on earth — was developed in the early 1960s by Pillsbury in collaboration with NASA and the U.S. Army Natick Laboratories to keep astronauts from being poisoned by their own food in space. It drew on Failure Mode and Effects Analysis (FMEA), the broader systems-engineering failure-analysis method the U.S. military had been using since the late 1940s. The same family of disciplines that ensured Apollo capsules did not have unacceptable failure modes was repurposed to ensure that astronaut food did not, and from there spread through the broader food industry. The food-safety profession has been doing systems-engineering failure analysis for sixty years. The cookbook is not making a novel connection. It is pointing at one that has been formalized longer than most of the people reading this have been alive.

Where the Analogy Is Imperfect

The strongest version of the case against this whole framing is that cooking and software differ in a way that should matter: cooking is one-shot, software is iterative. The argument lands cleanly when applied to software development — iterative improvement works for code in a way it does not work for remodeling your kitchen. Most software-engineering analogy work implicitly assumes development as the relevant activity, in which case the iteration argument wins by default.

The cookbook claims this argument is correct about development and irrelevant about production operations. Development is the kitchen design. Production is service. You can iterate on the kitchen design for years. You cannot iterate on Saturday night. A deployed email is a baked cake.

A second, more uncomfortable place where the analogy bends is that HACCP itself is prone to the failures it was designed to prevent. There is a long-running critique inside food-safety practice that HACCP systems fail when organizations implement the framework as paperwork without understanding the underlying tacit knowledge it was meant to formalize. This is the structural problem with all systems engineering applied to tacit-knowledge domains: the formalization is meant to compensate for tacit knowledge, but the formalization itself requires tacit knowledge to apply correctly. The recipe is not the chef. The runbook is not the engineer. The cookbook does not pretend otherwise. Every recipe in it has a technique note that points at the part the recipe cannot specify, on purpose. The thing the recipe cannot specify is the actual lesson.

A third seam is scale. Kitchens cap out at a certain size, beyond which they become commissaries with different physics. Production systems are still scaling exponentially. The properties hold across orders of magnitude, but the time available to react does not. CrowdStrike’s update reached 8.5 million machines before any human could form a complete sentence about what had happened. There is no kitchen analog to that detonation speed. The cookbook borrows the failure taxonomy; it does not pretend the response window is the same.

What to Do With This

Most readers will use this cookbook in one of two ways. Some will read it as a cookbook — make the dishes, eat them, notice that bread is hard for reasons that turn out to be the same reasons rolling deployments are hard. Some will read it as a postmortem collection — recognize the failure modes, transfer the pattern recognition into operations, notice that you hesitate the same way at the deployment terminal that you hesitate over the carbonara pan when the steam starts rising too fast. Both are intended. Reading it as both, simultaneously, is what the structural claim is for.

The practical insight, if you want one, is this: the part of any operational checklist that looks like “proceed with caution” is the part you are paying senior people to perform for you. Junior practitioners read the explicit steps. Senior practitioners read the implicit ones — the steps that say, in slightly different words: this is the part where you stop, look at the actual state of the system, and apply judgment that took five years to develop. When you write runbooks, mark those steps. Underline them. Tell people: this is the part where you call someone whose hand is calibrated. Do not let the runbook pretend it has captured the skill, because it hasn’t, and treating it as if it has is what gets you the carbonara that should have been a sauce.

CrowdStrike’s post-incident response named multiple engineering improvements — better content validation, enhanced error handling, customer-controlled deployment timing — and one process change: staggered rollout. The first three are explicit, mechanical fixes you can put on a checklist. The fourth is the implicit one. Staggered rollout is the operational version of “taste as you go.” You do not commit the entire batch before tasting a spoonful. The reason large engineering organizations sometimes ship updates that do not stagger is that the staggering step felt like overhead until the moment it didn’t, which is exactly the moment after which it is too late to add. That decision — when to taste, when to commit — is the tacit one. The runbook can list it. It cannot make it.

That is the cookbook in a sentence. The recipes are real. The failures are real. The thing in the middle — the thing the recipe cannot say, the runbook cannot say, and the postmortem only describes after the fact — is what both books are actually about.

Make the food. Read the failures. Pay attention to what neither one quite tells you.

Sources: CrowdStrike Channel File 291 / Falcon Sensor incident, July 19, 2024 (CrowdStrike Preliminary Post Incident Review and Root Cause Analysis); Microsoft 8.5 million figure (Microsoft blog statement, July 2024); Parametrix $5.4 billion estimate (Parametrix CrowdStrike Outage Impact Report, 2024); Knight Capital Power Peg incident, August 1, 2012 (SEC Press Release 2013-222 / SEC Order in Knight Capital Americas LLC); 2025 PubMed Central review of Maillard reaction mechanism; Polanyi, The Tacit Dimension (1966); 2025 Journal of Nutrition Education and Behavior paper applying Polanyi to cooking; Perrow, Normal Accidents: Living with High-Risk Technologies (1984); HACCP origins (Pillsbury / NASA / U.S. Army Natick Laboratories, early 1960s); FMEA origins (U.S. military, late 1940s); WannaCry, May 2017.