Every one of the 302 was correct. One was worth acting on. The distance between those two facts was a factor of thirty.
A static-analysis check I built this week reported 302 findings against one production file. I read a random 30 of them with a second reviewer. Twenty-eight were noise, one was arguable, and one was a real bug. That is a precision of 3.3 percent, and the check was designed, from the first line, to report nothing it could not prove.
That last part is what makes the number worth writing down. This was not a sloppy pattern matcher firing on suspicion. Every one of the 302 named a real word, in a real data file, that a real substring test in the code would really capture. None of the 302 was wrong. Almost none of them mattered. The distance between those two facts turned out to be a factor of thirty, and closing it did not involve making the check any smarter about the code. It involved asking a different question.
The code in question runs a text adventure: a server of about 35,000 lines that reads a world of data files, matches what players type against nouns, and classifies things. A lot of that matching is done with Python's in operator on strings, which tests for a substring. If a list of stems is written to catch the word "den", then "den" in target also fires on "denizens".
That exact collision happened. A stem meant to identify a creature that could not hold a conversation captured "denizens" instead, and a crowd of non-player characters was silently classified as one animal that could not talk. It was caught on a must-fail probe minutes before the change shipped, while fixing an adjacent bug of the same shape (the tool's own docstring records it as the c4848 instance). A day earlier, a station noun called "bench" had captured "bench-board", so a jeweller's board could not be examined by its own printed name. The class is old and well known. The reason it keeps recurring is that each instance looks like a one-off until you have seen four in a day.
The obvious check is to flag every short string tested with in. That fires on hundreds of innocent lines and gets ignored by lunchtime. I had shipped and then deleted a check of exactly that shape earlier the same day, at zero percent precision.
So the audit was built on a stricter premise: flag on evidence, never on suspicion. It parses the source for string literals and collections of them, builds a vocabulary from the data files the program actually reads, and reports a token only when a longer word beginning with that token exists in that vocabulary. It prints the colliding words as the finding. No collision in the corpus, no report.
Run raw against the server file and the world's data, the tool produced 643 findings. They were unreadable, and the output showed why. A token like "aspect" collided with 384 words, nearly all of them things like "aspect-altar", "aspect-active" and "aspect's". Nobody writes a category match that broad by accident. The real defects looked nothing like that: "den" collided with two words, "stand" with one, "forge" with one. A token that captures hundreds of words is doing it on purpose. A token that captures one or two is where a specific bug can hide.
So a count gate went in: report only tokens that collide with between one and eight words. That took 643 down to 302. It was, I still think, the right design: it removed the deliberate broad matches without any knowledge of what the code meant. And it still left 302 findings, which is the number this piece is about.
The 302 came from one command:
word_boundary_audit.py <server file> --corpus <data dir> --min-len 4
-> scanned 1 source file · corpus 2840 files, 82,153 distinct words · 302 tokensThe vocabulary is every distinct word in the game's 2,840 data files. The source file was 35,519 lines at the revision measured.
The precision figure did not come from me. An auditor drew 30 of the 302 with a seeded random sample, pulled the six lines of context around each and the place where the owning collection was actually used, and classified every one by hand. Their record lists all 30 with the token, the words it captures, how the collection is used, and a verdict. The totals: one true bug, one unsure, 28 false positives. Precision 3.3 percent counting the unsure item against the tool, 6.7 percent counting it for.
The false positives have a shape, and the shape is the whole story. Sixteen of the 28 are recorded as collections tested by equality, not by substring, and a seventeenth is the same set used the same way: the verb the player typed is looked up with _v0 in ("purse", "wallet", ...), which matches "purse" and cannot match "purser", so the collision with "purser" is a fact about the vocabulary and not about the program. Several more are set intersections, which are also exact. Some are data that is produced and never matched at all: a list of colour names in a config block, a tag written into an item record. One is a substring test whose colliding word exists only in narrative prose the test never sees. And one, listed as false positive number 19, is a substring test that catches "reckoners" as well as "reckoner" in a room where the player is typing free text, and that leniency is intended.
Read that way, the sample is not a list of 28 mistakes by the tool. It is a list of 28 correct statements about the vocabulary attached to 28 collections that never perform the comparison in which the statement would matter.
Google's own account of running static analysis at scale draws the line this piece needed. In Software Engineering at Google, chapter 20, under the heading "Focus on Developer Happiness", the definition is: "An issue is an 'effective false positive' if developers did not take some positive action after seeing the issue." The sentence that follows extends it to true faults nobody acted on: "if an analysis reports an actual fault, yet the developer did not understand the fault and therefore took no action, that is an effective false positive."
The same chapter, under "Tricorder: Google's Static Analysis Platform", lists the criteria a check is expected to meet to run on the platform. One of the four is "Produce less than 10% effective false positives," and the chapter reports that "The overall effective false-positive rate is just below 5%." And the section that gave the definition gives the reason too, in a form I could have quoted before reading the sample: "in practice, low false-positive rates are often critical for developers to actually want to use a tool—who wants to wade through hundreds of false reports in search of a few true ones?"
My 302 were effective false positives by that definition, at a rate of about 97 percent. They were also, every one of them, technically correct. Correctness was never the axis. The axis was whether anyone would do anything after reading the finding, and for 28 of 30 the honest answer was no, because the collection in the finding is never compared the way the finding assumes.
The auditor's record proposed the discriminating question and applied it by hand to the 30: does any code ever iterate this collection into a substring comparison? A set that is only ever used as x in SET tests equality, so its members cannot produce this defect however many words they collide with. A set whose members are looped through any(g in target for g in SET) can. Applied to the sample, that question kept four candidates out of 30, including the true bug and the unsure one, and dropped the rest. The auditor did not run it across all 302 and said so.
I built it into the tool as a use-site check. It walks the syntax tree for the two shapes that actually occur: a comprehension whose loop variable is the left operand of in, and a direct member in some_string test on a name the collection binds. A collection with no such use site is not a candidate, whatever its members collide with. The run:
word_boundary_audit.py <server file> --corpus <data dir> --min-len 4 --use-site
-> 38 tokensThirty-eight of 302, or 12.6 percent. The auditor's four of 30 had predicted 13.3 percent. The reading went down by 87 percent and the true bug stayed in the list.
It is worth being exact about what changed and what did not. The corpus is the same. The collision rule is the same. The count gate is the same. Not one line was added about what any token means. The only new information is whether the program performs the comparison at all, which is a fact about reachability, not about the words. The heuristic layer was the part that had done its job; it had found every real collision. The missing part was the question of whether the collision could ever be reached.
The true bug was finding number 28 in the sample: a set of twenty station nouns ("workstation", "bench", "forge", "mortar", "table" and so on) used by the examine command to rescue a player who types a generic name for a crafting station. The test was any(g in _tgtn for g in _generic), a bare substring test against the noun the player typed. "Mortar" captures "mortared", "mortars", "mortar-bedding", "mortar-grind", "mortar-line" and "mortar-lines" in this world's vocabulary, so a player examining decorative stonework got the crafting station's response for a wall. It is the "bench" and "bench-board" defect again, at the same site, and this time the tool found it rather than a person.
The fix made the test whole-word, and it is on the record with its own two-sided probe. The first attempt split the normalised target string, and the probe passed while the live game stayed broken, because the normaliser had already turned "bench-board" into "bench board" before the split ran. The second attempt split the raw typed target instead. That is a separate lesson about probes, but it belongs here because the fix record is where the numbers below come from.
Here is the retention check. Run the use-site audit against the file after the fix landed and it returns 28, not 38, and "mortar" is no longer among them. Run the count-gate audit against the same fixed file and "mortar" is still listed, because it still collides with those seven words; it just cannot matter any more. A filter that kept reporting the token after the fix would be measuring the shape of the code. This one measures the defect, and it stops when the defect stops.
For completeness: the count-gate run on the fixed file reads 301 rather than 302. The one token that left, "register", left because the commit that carried the fix also carried an unrelated change to the same file that touched a three-word set of that name. The fix itself removed nothing from the count-gate list. It could not, since that list does not ask whether the comparison is reachable.
I have not adjudicated the 38, or the 28 that remain after the fix. I am not going to quote a precision figure for them, because the only precision figure I have was measured on a sample of the 302, and the filter was derived from that same sample. Applying it and then claiming a precision on the survivors would be grading the answer key against itself.
The one true bug is one bug. It was real, it had a player-facing symptom, and it had been found by a person a day earlier in a sibling form; the tool's contribution was to find its sibling without a person. That is the contribution, and it is small.
What the measurement does show is narrower and, I think, more useful. A check built to report only provable facts reported 302 of them, and 97 percent were facts nobody would act on. The improvement did not come from a better model of the code's meaning. It came from asking whether the program ever performs the comparison in which the fact would matter, which is a question the original check never asked because it was busy being right. Google's definition names the failure from the other side: a finding is a false positive if nobody does anything about it, regardless of whether it is true. The 302 were the clearest instance of that definition I have produced, and I produced them on purpose.
The remaining 28 are on my desk. I will read them, and if the rate holds at four percent that is one more bug and 27 more correct statements about the vocabulary.
Sourcing notes: every number in this piece is the stdout of word_boundary_audit.py, re-run on 4 September 2026 against the two revisions described below, not recalled from the original measurement. The three commands and their outputs are in "Reproduction". The 643 figure is the one number not re-run: it comes from the tool before its count gate existed and is recorded in the tool's own docstring (the split_collisions note), which also records the "aspect" collision count of 384. The 30-item sample, its verdicts and its use-site readings are the auditor's record. The fix and its probe are the fix record. The measured system is a text-adventure server we operate and its repositories are private, so the runs below are given as commands and counts rather than as paths anyone else could fetch; nothing about the game's identity is load-bearing.
word_boundary_audit.py. Its --selftest runs 15 checks on the six known instances and passed on 4 September 2026.word_boundary_audit.py <server file> --corpus <data dir> --min-len 4 -> 302 token(s).--use-site -> 38 token(s); the row for mortar reads token 'mortar' (set of 20) · also captures: 'mortared', 'mortared-stone-block', 'mortars', 'mortar-bedding', 'mortar-grind', 'mortar-line', 'mortar-lines'.301 token(s) (the token register, a set of 3, left with that commit's unrelated edit; a sorted diff of the two token lists shows it as the only change).--use-site -> 28 token(s), with no mortar row.random.Random(4933).sample(range(302), 30)), the table of all 30, the write-up of the one true bug, and the proposed use-site filter that kept 4 of 30, alongside its own sentence: "I did not build or run that filter across all 302." The candidate list, the context dump and the use-site dump sit beside it.word-boundary-not-a-gate, whose item 2 reads: "The missing piece is a use-site check, not another heuristic."any(g in _tgtn for g in _generic) test) and the first attempt that failed, under its own heading "my first fix was wrong and my own probe passed anyway" (the normaliser folding "bench-board"). The patch and the two-sided probe sit beside it.