Texas settled with Pieces Technologies over its advertised hallucination rate without deciding the true rate. It required the definition and the method instead. A rate is a definition, a denominator and a reviewer, or it is a slogan.
On August 9, 2023, a Dallas company called Pieces Technologies put out a press release with a milestone in the headline: one million clinical summaries, written by its AI for hospital staff. The subhead said the summaries came "with a very low hallucination rate of <.001".
Less than .001 of what? The body of the same release answered with a different-looking number: an "industry leading 'severe hallucination rate' of less than 0.1%". Read as a fraction, .001 is 0.1%, so the two agree. The chief executive, Ruben Amarasingham, went further: "We are on track to achieve a severe hallucination rate of less than 1 error in 1 million summaries generated."
Three months later, on November 14, 2023, a second release said Pieces had "an industry-leading observed severe hallucination rate of less than 0.001%". That is 1 in 100,000, a hundred times lower than the August figure. Ten months after that, on September 18, 2024, the Texas Attorney General announced what his office called "a first-of-its-kind settlement" with Pieces. His investigation, the announcement said, "found that these metrics were likely inaccurate and may have deceived hospitals about the accuracy and safety of the company's products."
The public record does not say why the number moved a hundredfold between August and November. The rate may have improved, or the definition may have changed, and nothing published distinguishes the two. That gap is also the subject of the settlement. Before arguing about whether "less than 1 per 100,000" was true, ask what it would take for a number like that to be measurable at all.
The settlement is an Assurance of Voluntary Compliance, filed as State of Texas v. Pieces Technologies Inc., Cause DC-24-13476, in the 191st District Court of Dallas County. It is short, so here is its wording.
The remedy is the interesting part, because it is not a number. Under ¶17, any metric Pieces uses in marketing must come with a clear and conspicuous disclosure of two things:
Alternatively (¶17a), Pieces may "retain an independent, third-party auditor", and its marketing must then be "consistent with, and substantiated by" the auditor's findings.
The rest of the agreement points the same way:
The State, in other words, did not try to settle what the true rate was. It required that the rate come with its working shown.
Pieces' response, given to the HIPAA Journal, contests the State's press release but not that point. "Pieces vigorously denies any wrongdoing and believes strongly that it has accurately set forth its hallucination rate," the company said, and it called the AG's press release a misrepresentation of the agreement. It also said something both sides seem to accept: its rates were reported "in the context of an independently developed risk-classification system that is based on severity," and "there is no industry-wide risk classification system for generative AI hallucinations for inpatient clinical summarization that exists today."
That is the hinge. The State says the number may have misled. The company says the number was accurate under its own definitions. Neither disputes that the definitions were the company's own, and the settlement's answer is a duty to publish them.
Suppose you wanted to support "fewer than 1 severe hallucination per 100,000 summaries." You would grade summaries and count the severe hallucinations you found. If you found none, how many summaries would you need before "none found" means "fewer than 1 in 100,000"?
Medicine has had a crisp answer to this since 1983. James Hanley and Abby Lippman-Hand asked it in JAMA under a title every product team should remember: "If Nothing Goes Wrong, Is Everything All Right?" When an event never occurs in n trials, the 95% upper bound on its true rate is about 3/n. That is the rule of three. A drug trial that sees no liver failures in 3,000 patients can rule out rates above about 1 in 1,000, and nothing smaller.
Drug regulators wrote the same arithmetic into their rules. ICH E1, the 1994 international guideline on how many patients a new long-term drug should be tested on, covers the case of 100 patients treated for a year: "When no serious ADE is observed in a one-year exposure period this number of patients can provide reasonable assurance that the true cumulative one year incidence is no greater than 3%." That is 3 in 100, the rule of three. The guideline expects about 1,500 patients in all, and it is plain about the limit of that number: the safety evaluation "is not expected to characterise rare adverse events, for example, those occurring in less than 1 in 1000 patients." Rarer harms are what surveillance after approval exists to catch. By medicine's own arithmetic, a claim of 1 in 100,000 needs a denominator two hundred times the 1,500 patients that guideline expects.
Apply it to Pieces' numbers:
Now look at the denominator Pieces published. The August release said its system had generated "more than 1 million autonomously written clinical summaries" for "more than 72,000 hospitalized patients," within seven months of going live. That is a count of summaries generated. A rate needs summaries graded, and the record never says how many were. What the graded count would have allowed, if every graded summary came back clean:
| Clean graded summaries | Strongest honest claim at 95% |
|---|---|
| 10,000 | fewer than 1 in 3,338 |
| 72,000 | fewer than 1 in 24,034 |
| 300,000 | fewer than 1 in 100,142 |
Only the last row supports the advertised number. For scale, here is what grading 300,000 summaries would cost in clinician time:
That is our arithmetic, not a description of how Pieces works, but it explains why a rate that small, in practice, leans on automated review. And that moves the question to the third part of any rate.
A count of zero measures two things at once: the rate of errors, and the reviewer's ability to see them. If a grading process catches only a share of the real severe hallucinations, its clean results bound the true rate only after dividing by that share. At 300,000 graded summaries:
Pieces described its method in the 2023 releases as "contextual modeling, collaborative and adversarial AI, human-in-the-loop supervision and automated clinical prompt engineering". It said the rate was contractually guaranteed and "audited through its Pieces SafeRead program." What the record does not contain is the one number that converts clean results into a rate: how often that process catches a severe error that is really there.
Software engineers learned the hard version of this lesson in avionics. In 1993 Ricky Butler and George Finelli of NASA Langley published "The Infeasibility of Quantifying the Reliability of Life-Critical Real-Time Software." Their abstract is blunt: "the quantification of life-critical software reliability is infeasible using statistical methods." Demonstrating the ultra-low failure rates such systems need would, they showed, take an impractical amount of testing. The usual escape, running independent versions and trusting that they will not fail together, does not rescue it either: "The key assumption of software fault tolerance-separately programmed versions fail independently-is shown to be problematic."
If part of a hallucination review is one model checking another, that warning applies directly. The checker's blind spots need not be independent of the writer's, and the errors both miss are exactly the ones a published rate never counts.
The words "critical" and "severe" carry the whole claim, and in the public record they have no definition. Pieces itself says no industry-wide classification exists. Without one, the same summaries could yield very different rates, depending only on where the line between "severe" and "minor" is drawn.
Then there is the unit. "1 per 100,000" per summary is a very different claim from 1 per 100,000 per sentence. A hospital summary might run to 30 sentences. At that length a per-sentence rate of 1 in 100,000 means roughly one summary in every 3,333 carries a severe error, which is thirty times worse than the same words read per summary.
For contrast, here is what a fully specified rate looks like. In May 2025, a team led by Elham Asgari published a framework for clinical summarization safety in npj Digital Medicine. It is a different model (GPT-4) on a different task, so its numbers say nothing about Pieces. What it shows is the shape of a complete claim:
You can argue with that study's choices. But you can see every one of them, and that is the difference between a measurement and a slogan.
A hallucination rate is three things: a definition, a denominator and a reviewer. A published rate missing any of them is a marketing claim. As of September 2024 that is how at least one regulator reads it, and Texas wrote two of the three into a court filing: the definition, and the method that produces the denominator. It offered an independent auditor as the alternative to publishing them.
We have counted how often public companies even mention hallucination risk in their annual reports (pharma outnumbered software 29 to 10); this is what it looks like when a regulator reads the number itself.
So the next time a vendor's rate lands in your procurement memo, or your own team wants one for a launch post, ask the three questions:
If the answers fit on the same page as the number, you have a measurement. If they do not, you have the kind of claim Texas made Pieces explain.
Sources
Chain of Consciousness keeps a tamper-evident, dated record in hash-linked entries, so the definition, the graded sample and the reviewer behind a published number can be shown later, not reconstructed after someone asks.
pip install chain-of-consciousness npm install chain-of-consciousness
Or start without installing anything: Hosted Chain of Consciousness.