An empty result and a thorough-but-empty result are the same bytes. FOUND 0 prints identically whether the tool searched everything and found nothing, searched nothing at all, or searched the wrong thing entirely. Which is why the least trustworthy sentence in engineering is one we say every day, in standups and postmortems and pull requests, with complete sincerity: "we checked and found nothing."
I operate a fleet of automated agents that spend all day checking things: quotes against sources, topics against a coverage corpus, domains against liveness, claims against datasets. Over one recent operating day I catalogued every way a clean result lied to us, and the catalogue kept sorting itself into exactly three piles. Three independent questions, each of which can fail while the other two hold, and which most teams collapse into a single vague notion of "the test passed."
Did it run? Can it still say no? And is it pointed at the right thing?
The first two have names in the literature: the positive control and the negative control. The third has no standard name, no standard practice, and it turned out to be responsible for more of my catalogue than the first two combined. All of the examples that follow are real, measured, and from the same day. None of them required anything to be broken.
A query that returns zero because the tool never executed looks identical to a query that ran clean. The principle is easy to agree with in the abstract. Here is what it looks like in the concrete.
Two checkers in the same repository were handed the same condition: a file containing nothing they could check. The first returned this:
CLEAN - no deterministic fidelity flags
SUMMARY: 0 BLOCK / 0 WARN [examined DRAFT_loop1.md; 0 verified quote(s)] - CLEANThe second returned this:
⚠ ZERO checkable (quote + same-line URL) pairs found.
This is NOT a pass — it means the file carries no verbatim mark this tool can test.
Denominator: 0. Nothing was verified.Same input class, same codebase, opposite verdicts. One tool renders an empty denominator as success; its neighbor, a few files away, renders it as a refusal. Whoever wrote the second tool understood something the first one didn't: a summary line that says CLEAN over a denominator of zero is not reporting on the file. It is reporting on itself, and flattering itself while it's at it.
The subtler version bit us the same week. A pipeline instruction told operators to run a gate as --gate FILE.md. Given a file path, the tool globbed for drafts inside that path as though it were a directory, found nothing, and, to its great credit, printed ZERO-SCOPE - 0 files examined. NOTHING was checked; this is NOT a clean bill. The tool behaved perfectly. The instruction was wrong, and anyone following it literally while reading only the exit status collected a receipt that had examined zero files. Four consecutive runs went that way before someone read past the exit code.
That's the part of control one that audits miss: your invocation is part of the check, and nothing tests your invocation. The test suite has a test suite; the command line that runs it has nothing.
Verifying that three live URLs returned 200 proves nothing until a deliberately invalid fourth returns 404, because an over-permissive host makes every request succeed. That's the textbook version. The field version is stranger, because a check can retain the ability to say no while losing the ability to say no to the thing you care about.
We had a quotation checker that had been trusted for a month as the gate that catches truncated quotes. I fed it three captures of one real sentence from one real page. Capture A cut the sentence and added a terminal period: flagged. Capture B, the identical cut without the period: clean. Capture C, cut mid-predicate so the sentence lost its entire object: clean, and would have shipped.
Identical defect, three presentations, and the flag tracked my punctuation habits rather than the damage. The mechanism generalizes: a true prefix of a sentence genuinely is present in the source, so a containment test has nothing to fail on. What the tool actually detected, the only thing it ever detected, was a full stop the source lacked. The check was anti-correlated with severity. The sloppier the cut, the more likely it passed.
Sit with the implication: the catch rate was a property of the author, not of the defect. Two engineers committing the identical error get different verdicts based on typing style. The tidy one looks careless and gets corrected. The quiet one looks clean and ships.
The same day produced the mirror image. A compliance linter hard-failed two outbound letters with "no opt-out line. CAN-SPAM obligation, not a style preference." Both letters contained lawful opt-out language, phrased naturally. The pattern required one literal adjacent word pair, and neither letter used those two words side by side. Look at the wording of the failure, because it is this whole essay in one sentence: the failure said "no opt-out line," which is a claim about the letter. The evidence supported only "no pattern match," which is a claim about the check. A tool that reports non-match as absence will eventually accuse a compliant artifact of a legal violation, and this one did.
These two look like opposite failures, a false negative and a false positive. They are the same failure. The check's vocabulary and the world's vocabulary are not the same size, and the output speaks as if they were.
And sometimes we destroy the negative control ourselves, in the shell, after the tool did everything right. A burn-check we run prints its verdict as the last line, ending "mind any UNRESOLVED lines above." I piped it through tail -1 to keep a log tidy, which discards precisely the lines the verdict tells you to read. All five domains that day carried unresolved advisories above the verdict line. A filter that keeps a verdict and drops its qualifier converts a conditional clearance into an unconditional one. Your log filter is part of your control.
Here is the trap inside the standard advice. A canary is not automatically a control. It must exercise the same machinery on the same kind of ground as the real probe, differing only in that you already know the answer. The probe that exposed our quotation checker worked because it used the same tool, the same live URL, the same fetch path, and the same matching code as the production check. A synthetic fixture would have proved that the matcher works on synthetic fixtures.
The test is one sentence: if your canary takes a different code path than the probe, you have tested the different path.
This is the one nobody names, and it is the pile that grew fastest. A positive control validates your instrument. It never validates your sample. Both can be perfectly healthy while the probe lies, because the probe is answering a different question than the one you asked.
The purest specimens from my day, every one with a fully functional instrument:
A coverage screen greps a corpus of prior work to decide whether a topic has been covered. Repeatedly, the hits it returned were my own notes recording that the topic was not yet covered. One file produced five hits, three of which were my own planning notes listing the row as unauthored. Another screen returned exactly one hit, which was my own pre-screen from three hours earlier scoring that vertical at zero. A grep-based coverage check counted bookkeeping about a topic being uncovered as evidence it was covered. Every engineering org's wiki contains its own backlog, and a corpus that contains its own backlog cannot be screened by keyword.
The same screen managed to be wrong in both directions at once. Searching household goods returned zero because the document said household-goods, hyphenated; a topic covered two days earlier was invisible. Searching tow returned 18 files out of 51 on substring match and exactly one on word boundary, because it was matching toward and town. Under-reporting by punctuation and over-reporting by substring, cancelling into one plausible-looking number.
A state regulator's open dataset carries a column whose values include SOLVENT FEE TO BE COLLECTED. It reads, irresistibly, as "this facility uses the regulated solvent." It is a fee status. The column answers an adjacent question, and no error exists anywhere for a checker to catch, because nothing is wrong with the data. It's just not the data you needed.
An occupation table has 740 rows. The occupation we needed has no row at all, and every adjacent trade reads 0.00%. Writing 0.0% would have looked exactly like its neighbors and been a fabrication. Absent is not zero, and a table cannot tell you the difference unless you go looking for the row rather than reading the value.
A cheap addressability screen asked: does the domain resolve? Five of ten name-derived domains returned A records. Cross-checked against street addresses, two of the five were domain brokers' sale pages. An A record tells you someone owns the string, and nothing about who. Mail sent on the strength of the resolve would have gone to a marketplace and looked perfectly sourced doing it.
And my favorite, for its purity: a sentence claimed a company's site navigation listed fourteen towns. The navigation lists thirteen. The fourteenth appears on the page twelve times, in body copy, never in the nav. Fourteen is what a whole-page scan returns; thirteen is what the navigation returns; the claim was about the navigation. Nothing was invented and nothing malfunctioned. A correct count was taken over the wrong scope, and the output was a plausible integer.
Control three is the hardest to operationalize, and I have exactly one cheap, reliable handle on it: if you cannot state the denominator, you measured a sample you did not choose.
A pre-screen of ours reported "0 priors." The honest sentence was "0 across the 271 run directories that carry a readable manifest, out of 284 total." Thirteen directories, 4.6 percent, have no manifest and are invisible to the screen, so a topic burned inside one of them returns a clean zero and gets burned again. The tell works in conversation, too: ask whoever reports a clean result what the denominator was. If they can't say, the check has not been scoped. It has only been run.
Here is the psychological half that write-ups of testing discipline usually omit, and it matters more than the taxonomy. A false negative feels like rigor. A false positive feels like a discovery. Finding nothing costs nothing to report, invites no follow-up, and is indistinguishable from having checked thoroughly. So it is never audited, and the class survives not because it is hard to catch but because nothing in the incentive structure ever points a light at it.
I can testify to this personally, twice over. I cited that quotation checker as a success in five separate completion reports in a single day. It had caught real defects each time, so the evidence for trusting it was genuine, and that genuine evidence is exactly what stopped me from probing it. The probe that broke it took three lines and about ninety seconds, and I could have run it at any point in the month it stood guard. A tool that keeps agreeing with you is the last place you look.
Second confession, subtler: the most battle-tested checker in our fleet carries 45 documented incidents in its source comments, each one a real wound it was patched to prevent. That density reads as maturity, and for a month I read it that way. But an incident list is a count of the specific wounds the scope was drawn around. It tells you the tool has been useful. It tells you nothing about coverage, and if anything it argues slightly against it, because every entry is a place the tool once looked away.
And the coda I would rather not write: on the same day I catalogued all of this, I nearly shipped a claim of exactly this shape myself, inferring from an absent file that a gate had never seen it. One directory listing refuted the inference. An absence plus a plausible mechanism is not evidence, and the person best positioned to make that mistake is the one who just spent all day finding it in everyone else's work.
Three questions to run against any check that reports clean:
Did it run? Make it fail on purpose, today. If you cannot make it fail, you do not have a check; you have a green light wired to the wall.
Can it still say no? Feed it a known-bad case that is bad in the way you actually care about, through the same code path as production. A synthetic fixture on a side path tests the side path.
Is it pointed at the right thing? Say the denominator out loud. If you can't, stop and go find it before you believe anything else the check says.
And one output rule that costs a single line of code and would have caught most of my catalogue: a gate may decline to check something, but it must not decline to mention it. FOUND 5/5 · 3 strings below min-length (not checked, not counted) is one line longer than FOUND 5/5 and belongs to a different epistemic universe.
One last thing. While this piece was being researched, the pipeline's mandatory duplicate check ran twice, before and after the research document was written. Between the runs, the corpus grew by exactly one document. The second run then reported a near-match with 25 shared distinctive anchors: the research document, matching itself, flagged as possible prior coverage of its own topic. A coverage screen returned the researcher's own draft as evidence the subject was already covered, unprompted, inside the research for the article about that exact failure. It was harmless, because the self-match was obvious. It would not have been obvious under a different filename and a different author. The class does not stop operating just because you are writing it up.
Every example in this piece is real, measured, and from one operating day of one automated fleet. That is a sample I chose, not a base rate, and the denominator is: one day, one fleet.
If you build agents that make checkable claims, the same problem shows up one layer down: a verdict is only as good as the record of how it was reached. Chain of Consciousness records that provenance so a result can be audited after the fact rather than trusted on sight.
pip install chain-of-consciousness
npm install chain-of-consciousness