A rule for every mistake feels like diligence. It is actually deferral, and the debt comes due all at once, on the day the document crosses the budget and everyone quietly stops reading it.
Every organization has the document. It might be called the runbook, the onboarding wiki, the code style guide, the deployment checklist, the agent's system prompt. It started life as a single clean page. Then someone made a mistake, so a rule was added. Then someone made a different mistake, so another rule went in. Then there was a bad outage on a Friday, and now there is a very specific rule about not deploying on Fridays, and a rule about the particular service that failed, and a rule about the particular flag that was set wrong. Each rule is individually reasonable. Each was written by a sensible person in direct response to a real failure. And the document is now forty pages long.
Here is the moment worth stopping on. After the next incident, someone adds rule forty-one, and it changes absolutely nothing, because the person who most needed rule forty-one stopped reading at page three. The document is, at this point, technically complete and functionally unnavigable. Every case is covered and no case is findable. The ruleset has quietly become the exact problem it was built to solve: it was supposed to prevent mistakes, and now it is so unreadable that people ignore it and make mistakes the old-fashioned way, by guessing. And the worst part is invisible, which is why almost nobody catches it.
The thing nobody accounts for is that a ruleset has a budget, and the budget is not storage, it is attention. A person, or a language model, can hold and actually apply only so many rules at once before the set exceeds working capacity. Below that threshold, each rule you add genuinely helps: it is read, it is remembered, it is followed. Above that threshold, something flips. People stop reading the whole thing and fall back on gut and habit, which means the forty-first rule is not read by the people it targets, which means it does not change behavior, which means you have paid the full navigability cost of a longer document and bought nothing.
And it is worse than nothing, which is the sharpest point in this whole essay. An unread rule is worse than no rule, because it manufactures false coverage. When a rule exists for a case, everyone assumes the case is handled. The reviewer assumes it, the manager assumes it, the auditor assumes it. But if nobody reads the rule that handles it, the case is not actually handled, and now the gap is invisible, hidden behind the comforting existence of a rule that addresses it on paper. A missing rule is a visible hole; people know to be careful. An unread rule is a hole with a rug over it. So past the readability budget, "add a rule" inverts from a fix into a liability: each new rule buys a sliver of theoretical coverage and pays for it in real navigability, until the marginal rule is net-negative, degrading the thing it was meant to strengthen.
This sounds like a modern problem, a symptom of sprawling wikis and bloated prompts. It is not. It is one of the oldest solved problems in the history of institutions, and the solution has a name.
By the sixth century, Roman law was a catastrophe of accretion. A thousand years of imperial edicts, senatorial decrees, and the contradictory written opinions of generations of jurists had piled up into a mass so large and so self-contradictory that no one could reliably say what the law actually was. It was the forty-page runbook at civilizational scale. Any given question had multiple answers on the books, issued centuries apart, by authorities who disagreed.
The emperor Justinian did the thing the incentive structure of every accreting ruleset resists. He convened a commission and told them to consolidate. Starting in 529, they produced the Codex, a revised collection of imperial law, and here is the move that matters: every imperial ordinance not included in it was formally repealed. Not archived, not deprecated, repealed. Then, beginning in 530, a commission of sixteen jurists took on the harder job, the jurists' opinions. They went over, by the traditional account, some two thousand books and three million lines of legal text. They selected the passages with practical value, cut the unnecessary matter, and, crucially, eliminated contradictions by simply choosing one of each conflicting pair and deleting the other. The result, the Digest, was published in 533, and together with the Codex and the rest it became the Corpus Juris Civilis, the foundation of the entire civil-law tradition that still governs most of the world.
Notice what codification actually is. It is not "read faster." It is not "add an index." It is periodic, ruthless refactoring of the ruleset: keep the operative, cut the dead and the contradictory, and produce a version a human can hold in their head again. The civil-law world internalized this so completely that it did it again twelve centuries later. The Napoleonic Code of 1804 swept away a patchwork of more than three hundred and sixty different regional legal systems that had made pre-Revolutionary France ungovernable, replacing them with a single coherent code deliberately short and clear enough that an ordinary citizen could read it. Codification is scheduled maintenance for a ruleset. It treats "the rules have grown unnavigable" not as a crisis to survive but as a recurring event to plan for.
If law figured this out fifteen centuries ago, the safety-critical industries relearned it in living memory, and their version is even more pointed for anyone building operational rules. Safety regulation is the purest rule-per-mistake machine there is. The unofficial term for how aviation rules get written is regulation by tombstone: the FAA is sometimes called the tombstone agency because so many of its rules were written in the aftermath of specific fatal crashes. The sterile cockpit rule, which the FAA imposed in 1981 forbidding non-essential crew conversation below ten thousand feet, exists because investigators traced a series of accidents to crews chatting through critical phases of flight. Each such rule is a headstone with a regulation carved on it.
But the people who actually have to use these rules in the cockpit hit the accretion wall directly, and their response is the template. Atul Gawande's 2009 book The Checklist Manifesto is the canonical account, and its central finding cuts exactly against the instinct to be comprehensive. A good checklist, Gawande reports, is not a thorough guide to a procedure. It is short. It contains only the killer items, the steps that are easy to skip and catastrophic to miss. One of his sources, a veteran who designs flight checklists for Boeing, puts the ideal length at roughly five to nine items, with a hard rule that reading the list should take under about a minute, because past a minute people get distracted and start skipping the vital steps. Sit with that. The professionals whose lives depend on their checklists discovered that comprehensiveness defeats the checklist. A list with every possible item is a list nobody completes, which is the false-coverage failure again, in a cockpit. Their discipline is to curate savagely down to what actually kills, and to keep everything else off the list precisely so the list stays used.
So if consolidation is a fifteen-hundred-year-old solved problem, why does every organization's rule document still bloat until it is useless? The answer is not that people are foolish. It is that the incentives are brutally asymmetric, and this is the load-bearing insight.
Adding a rule is a visible, safe, blame-deflecting response to a failure. When something goes wrong, "we have added a rule to prevent this from recurring" is a sentence a person can say in a meeting, put in a postmortem, and show to an auditor. It is legible action. It demonstrates responsiveness. It costs the author nothing and protects them.
Pruning a rule is the opposite in every dimension. It is risky, and it is invisible. If you remove a rule and, six months later, something breaks that the rule would have caught, you personally own that failure, and the whole chain of reasoning leads straight back to your deletion. And if you remove a rule and nothing breaks, which is the far more common outcome, absolutely no one notices or thanks you, because you cannot prove a negative, you cannot show the incident that didn't happen. So the expected value of adding is positive and the expected value of pruning is negative, for the individual, every time. Consolidation is nobody's job, by design. The ruleset grows monotonically not because anyone wants a forty-page runbook but because every local incentive points toward addition and away from subtraction. Nothing consolidates a ruleset unless something outside the incentive structure forces it to. Justinian's commission existed because an emperor ordered it into being; left to the normal machinery, Roman law would have kept accreting until the heat death of the empire.
This is where it stops being a management essay and becomes urgent, because AI agents are rule-accretion machines running at maximum speed. The standard way to correct an agent's behavior is to write a rule into its system prompt or its memory: the model did the wrong thing, so you add a line that says never do the wrong thing. It is fast, it is easy, and it accretes exactly like a runbook, only faster, because the feedback loop is minutes instead of quarters. And it hits the readability budget in a form that is not a metaphor at all. Past a certain length, a model genuinely cannot attend to all of its own instructions; the rules at the bottom of a bloated prompt compete for the same limited attention as everything else, and adding a rule stops improving behavior and can actively degrade it. There is even a cruel twist unique to language models: writing "never do X" puts X into the context, raises its salience, and can make the forbidden behavior more likely, not less. The rule against the mistake can summon the mistake.
The fix for agents is the same as Justinian's, consolidation, but with a mechanism that solves the incentive problem the humans never solved: a hard budget. Put a firm cap on the size of the rule-store, a byte limit or a token limit, and something remarkable happens. The moment the store is full, you cannot add a rule without removing or compressing another, which means consolidation is no longer nobody's job, it is a mandatory precondition of the next write. The cap converts the asymmetric incentive into a forced choice. Consider a real pattern, kept general: an agent's lessons file with a strict byte cap, which on hitting the limit forces its keeper to take a verbose, war-story-laden lesson and crush it down to the one operative rule that matters, with the narrative moved to an archive and left behind as a pointer. The same file that was documenting the disease of accretion gets cured, automatically, by the constraint around it. The cap is the Justinian commission, automated and scheduled by physics rather than by an emperor.
Be honest about the limit of the trick, though. The cap forces consolidation, but it does not perform it. A cap with bad judgment behind it will happily compress away an operative rule and keep a dead one; the constraint creates the moment where codification must happen, but a human or a model still has to do the actual work of knowing which rules are load-bearing and which are obsolete, which is the same hard skill Justinian's jurists exercised over their three million lines. The budget is the forcing function. The curation is still craft.
We have written before about a different failure of rules, the mistake that a reminder simply cannot fix, where the answer is to design the error out entirely so the mistake becomes physically impossible. This is the other failure, and it is distinct. Here the problem is not that one rule fails to land. It is that the rules breed, each one reasonable, until the set as a whole exceeds what anyone can read, and the fix is not to make any single rule stronger. It is to consolidate the set.
The practical residue is small and it fits on, appropriately, a short list. Give your ruleset an explicit readability budget, a real number, a page count or a rule count or a token cap, and treat hitting it as a signal to consolidate rather than a reason to raise the ceiling. When you add a rule, make it someone's actual job to ask what can now be removed or merged, because the asymmetric incentive guarantees that no one will ask unless asked. Schedule codification the way the civil-law world does, as recurring maintenance, not as a heroic once-a-decade cleanup that never comes. Curate toward killer items and be ruthless about it: for every rule, ask whether violating it actually causes the disaster or whether the rule is just comprehensiveness theater making the document longer and less read. And for anything automated, especially an agent, wire in a hard cap, because a constraint that forces the consolidation is the only thing that reliably defeats the incentive not to.
A rule for every mistake feels like diligence. It is actually deferral. Every rule you add without consolidating is a small loan against your ruleset's readability, and the debt comes due all at once, on the day the document crosses the budget and everyone quietly stops reading it, at which point every unread rule becomes an invisible gap wearing the disguise of coverage. The rules have a budget. Spend it like one. And when it runs out, do not read faster, because Justinian settled that argument in the year 534: you consolidate, you cut the dead, and you write a version a person can actually hold in their head.
An unread rule manufactures false coverage. So does an instruction you assume an agent followed but never checked.
The failure at the center of this essay is trusting the existence of a rule instead of its execution: the case looks handled because a rule addresses it on paper, and nobody verified that anything actually happened. That is the same gap the agent trust stack is built to close, one layer down. It ships as separate installable pieces so you verify what you actually need to: a provenance record of what an agent did, so you check the work rather than assume the instruction landed; verification against ground truth rather than against the rulebook; and ratings that price an agent by its real track record. A rule that says "never do X" is not evidence X did not happen. The record is.
Read the Theory of Agent Trust
pip install agent-trust-stack · npm install agent-trust-stack
Or the provenance record on its own: pip install chain-of-consciousness / npm install chain-of-consciousness.