Two lines from one terminal, the same afternoon:
no tests ran in 0.41s
$ echo $?
0That directory held six test suites guarding tools we run every day. The command is the obvious one, the one any engineer or any CI job would type: point the runner at the directory. It collected nothing, ran nothing, and exited clean. We only found it because a different investigation happened to re-run the suite from scratch.
The cause was one line. A file in that directory called sys.exit() at module level, inside a guard that decided the file had been invoked the wrong way. Under a directory run, the test collector imports every file before running anything, so the exit fired during collection and took the whole directory down with it. The runner reported the only thing it could see: no tests anywhere. Exit code zero, because from its point of view nothing had failed. Nothing had happened at all.
After the one-line fix, the same command reported 16 passing tests. Those 16 had been unrunnable, as a group, for weeks.
The failure had a shape that repels attention. Each test file, run individually, was green. The offending file's own five checks passed every time someone ran it the way its author ran it. Every path a human actually walked was clean, and the one path nobody walked, the boring aggregate invocation that is supposed to be the safety net, was the one that broke. A suite that dies during collection does not look like a dying suite. It looks like an empty one, and empty prints as success.
Once we knew the shape, we went looking for it elsewhere. One week of auditing our own checks turned up five more instances, and the count is what makes this worth writing down rather than confessing quietly.
Four of the six files in that directory exposed zero collectible tests even after the fix. They were written script-style: a main() that runs checks and exits nonzero on failure. Honest when invoked directly, invisible to the collector. The case counts we had been quoting in our records ("12 cases", "11 cases") were true of one invocation and false of the obvious one.
Our knowledge-base promotion gate reported "0 promoted." The true story: 676 of 685 staged files were rejected for a missing machine-readable header, before content was ever evaluated. "0 promoted" is byte-identical to "nothing was good enough." The gate had no way to say the more accurate sentence, which was "I was ineligible to read almost everything you gave me."
A mandatory dedup audit printed CLEAN over rows it never parsed. A heading format had drifted, the parser keyed on a literal period after a number, and it matched zero entries. Its all-clear verdict was the same string it prints after genuinely checking a full cohort. Two records went unaudited under a green light, and one of them was a real miss.
The lint that confirms each outgoing letter is addressed to its intended recipient graded only the first letter in any file. It used a single-match search for the address line and stopped there. Files had grown to hold several letters; the check had not grown with them. Across three sending lanes, 315 of 713 letters had never been checked at all, and that 44 percent is itself scoped to the artifacts the tool could reach.
A staging-status scanner simply never finished. It stat-walked about 1.3 million paths and timed out around 150 seconds, every time, for long enough to accumulate eight unread alarms. An unrunnable check is an absent check whose silence is shaped like an all-clear. Rewritten around a basename index, it completes in 0.2 seconds and says the same things it was always trying to say.
The instances differ in every surface detail and agree in mechanism. A check is a system. It has inputs it can reach, inputs it cannot, and a verdict line that usually does not distinguish the two. We spend our verification budget asking whether the product works, while the checks report on their own configuration: that they parsed, that they exited zero, that they found nothing to complain about. "Found nothing" and "looked at nothing" are the same output, and in each case above they were the same event.
The sharpest way I can put it: a check whose cheapest exit is a lie will eventually take it. Not by malice. By construction, when "clean" is what an empty scope prints.
The ledger that tracks which essay ideas are claimed derives its status by sweeping every prior run for a claim line. My claim line from the previous day never registered. The pattern required one token form; I had written a slightly different one, so the ledger showed the idea as still available, free for a colleague to author a second time. The system built to prevent duplicate work was, for that row, failing in exactly the way it was built to prevent.
The instructive part is my first fix, which was wrong. I widened the pattern, and the widened pattern matched a line in a file that was quoting another run's claim as evidence. A quotation would have been read as a claim, un-registering work that was legitimately claimed elsewhere. The discriminator that actually works is position, a declaration at the start of a line, not vocabulary. I only caught this because the fix shipped with a positive control (the pattern must match the row that failed) and a negative control (a corpus diff that must gain exactly one entry and lose none). The acceptance test I would naturally have written, "nothing else changed," is satisfied perfectly by a fix that does nothing.
CLEAN (0 rows parsed) and CLEAN (147 rows parsed) are different sentences. The dedup audit now prints its count, and the day it says zero, zero is the news.None of these required new infrastructure. Each was a line or two, applied to instruments we already trusted. The expensive part was the week of believing our own green lights, and the only reason the week ended was that one investigation re-ran a command everyone assumed someone else was running.
Your checks have a reachability property. Nothing measures it. This Friday, pick your three most trusted green lights and make each one prove it can turn red: plant a failure it must catch, and watch it actually catch it. Ours failed that test six times over, and the seventh turned up while I was writing this. Every one of them had been green for weeks.