A clean process exit never meant the work was done. A present, plausible artifact does not either.
Here is a three-line experiment I ran on my own machine this morning. It is the oldest lie in computing, and it still works:
$ false | tee report.txt
$ echo $?
0The pipeline exited clean. The file exists at the expected path. And report.txt is zero bytes, because false (standing in for your curl that just got a 404) produced nothing, and tee faithfully, successfully wrote the nothing down. Every downstream system that checks "did the command succeed, does the file exist" advances. This is not an exotic bug; it is the specified default of the POSIX shell: a pipeline's exit status is the last command's status, so everything upstream of the final pipe can die silently. The fix, set -o pipefail, lived in shells as a nonstandard extension for decades, and was only admitted into the POSIX standard itself in the 2024 revision (POSIX.1-2024, Issue 8). Unix pipes shipped in 1973. It took half a century for "the pipeline said it was fine" to officially stop meaning fine.
We have written before about one half of this disease. In "'Done' Is Not a State", we argued that a task that completed and a task that was silently dropped look identical from the outside (both just go quiet), so no system that infers completion from the absence of signal can tell success from abandonment. That essay's prescription stands: completion has to be an explicit, owned state, not an inference.
This essay is the uncomfortable sequel, and it starts with a confession: the working brief for this piece proposed the natural next fix. Don't infer completion from silence, check the artifact. Does a file exist at the expected path? Is it non-empty? Is it in the expected language? Advance only then. Reasonable, cheap, mechanical. It's the fix everyone reaches for second, precisely because the first essay's lesson landed: if quiet can't prove done, then surely evidence can, and an artifact sitting at the agreed path looks like evidence. The whole orchestration tradition points this way; it's the same instinct that made explicit completion states and artifact-existence sensors the respectable answer to ghost state in classical pipelines.
And it has the same bug. My three-line demo above defeats two-thirds of it from 1973. And the moment your pipeline workers are LLMs, the remaining third collapses too, because a language model is a machine for producing artifacts that are present, non-empty, fluent, on-format, and empty of work. Non-empty is the new exit zero.
The classical failure that "check the artifact" guards against is ghost state: the pipeline advanced, but the work's output is missing. An orchestrator marks a step done; the file was never written; everything downstream starves. Artifact checks genuinely catch this, which is why they feel like the fix.
But LLM workers introduce a strictly harder failure, one the batch-job world rarely produced: the artifact is there, and it's wrong in a way that carries no mechanical signature. Ask a model to summarize forty documents when retrieval quietly returned three, and you don't get an error: you get a confident, well-structured summary of three documents wearing the byline of forty. Ask for an analysis and hand it a truncated input, and it reasons beautifully over the truncation. Practitioners who run these systems keep converging on the same description: LLM failure is silent, the model produces confidently wrong output with no error state, and the failure is invisible unless someone downstream already knows what the right answer looks like. Schema and structure checks can't help you here, because they are deterministic answers to a different question: they know whether the output parses, and nothing about whether it means anything.
Line the two failures up and the upgrade is obvious. The old ghost was an absence: expected file, not there, trivially detectable, which is why a one-line existence check kills it. The new ghost is a presence: expected file, right path, right shape, plausible prose, no work inside. Every check the brief proposed (exists, non-empty, right language) passes. A perfectly camouflaged failure, produced not by an adversary but by the ordinary operation of a system whose defining talent is generating plausible text.
The bitter symmetry is what makes this worth an essay rather than a bug report: each generation's completion check fails the same way. exit 0 was a proxy, "the process didn't report failure" standing in for "the work happened." The artifact check is a better proxy, "something plausible landed where output belongs." But it's still a proxy, and it fails in the same shape: a cheap mechanical signal, necessary but not sufficient, that success and a specific class of failure both satisfy. We didn't eliminate the confusion. We moved it up a level, where it's harder to see.
The industry's actual response to this, visible in every serious writeup of LLM pipeline validation, is a ladder of checks, each catching what the previous rung waves through:
Every rung is real. You should build most of them; they're cheap, and each one converts a class of silent failure into a loud one. Rung 5 would have caught the summarize-three-as-forty ghost ("output must reference ≥ 35 source documents"). Rung 3 catches truncation. This ladder is simply what engineering maturity looks like.
But look at the top rung honestly, because there's a trap in it. The only check that addresses the LLM ghost directly (did the work actually happen, is this meaningful) is a semantic judgment, and at pipeline scale that judge is another LLM. Which reintroduces everything you were defending against: a stochastic component whose own failures are fluent and silent, checking a stochastic component whose failures are fluent and silent. Same-layer checking. The judge can be lazy, biased toward plausible-sounding text (it is the demographic that produces plausible-sounding text), or wrong with confidence, and now its verdict wears the costume of verification.
This isn't an argument against judges; used well, with tight rubrics and calibration against known-good and known-bad examples, they raise the bar substantially. It's an argument against the fantasy that some rung of the ladder finally bottoms out in certainty. It doesn't, and it can't, because every rung is a proxy for a fact ("the task the human cared about is done") that lives outside the pipeline entirely. There is no mechanical check that terminates the regress. There are only checks that push the ambiguity somewhere it's less likely to hurt.
Once you say it that way, the design question changes. It stops being "which check proves completion?" (none of them do) and becomes "where do I want the residual uncertainty to sit, and at what price?" That question has actual answers.
1. Verify where the work lands, not where it's written. The strongest completion signals aren't inspections of the artifact, they're observations of the world the artifact was supposed to change. Don't check that the migration script's output file looks right; run the query the migration was supposed to make fast. Don't grade the generated code by reading it; execute its tests. Don't audit the summary's word count; check that the claim extracted into the database matches a spot-checked source. Ground truth is expensive and sometimes slow, which is exactly why it's trustworthy: it's the one check that isn't a proxy, because it is the fact the proxies were standing in for. You can't afford it everywhere. Buy it where the blast radius is.
2. Prove your gate can fail. A completion check you've only ever seen pass is itself an unverified claim; maybe it's rigorous, maybe it's exit 0 in a nicer costume, and green-forever looks identical either way. So bait it, with a small standing family of poisoned artifacts, each one targeting a specific rung:
tee demo, bottled): should die at the non-empty rung;Run the family on every gate change, the way you'd run a test suite. Every red is a class of silent failure converted into a loud one; every bait that sails through green is your gate telling you, cheaply and in private, exactly which lie it will believe in production. pipefail is the pattern in miniature: it didn't make pipelines honest, it made one specific lie impossible. That's all any gate ever does, and a gate that can't demonstrate which lie it prevents should be presumed to prevent none.
3. Route the residue deliberately. After the ladder and the baiting, some ambiguity remains: the judge might be wrong, the spot-check might miss. The failure mode is pretending otherwise; the design move is budgeting it. Irreversible, outward-facing, or expensive actions get the costly verifier: a human, a ground-truth execution, a second independent judge with a different prompt and model. Bulk, reversible, internal steps get sampling audits and cheap rungs. What you never do is let the residue pool silently at the most dangerous step because every step's dashboard was green. Uncertainty is like any other load: it goes where you route it, and if you don't route it, it goes everywhere.
To make the whole thing concrete, here is what an honest completion predicate looks like for one boring real step, "summarize this week's support tickets for the exec digest":
exit 0 # the worker didn't crash
digest.md exists, > 0 bytes # tee can't fool us
parses as markdown w/ 5 sections# structure
cites >= 90% of ticket IDs # content rule - kills three-as-forty
judge: rubric score >= 4/5 # semantic, calibrated on 10 known digests
LANDING: 3 sampled claims # ground truth - traced to actual tickets
baits: empty / filler / 3-as-40 # the gate proves it can fail, weeklySeven lines. The first five are proxies and say so; the sixth is the world; the seventh keeps everyone honest. Nothing in it is clever. The only design decision that matters is that no single line, especially not the judge, is allowed to mean "done" by itself.
The half-century between | and pipefail-in-the-standard is not a story about shell maintainers being slow. It's a story about how long a cheap, almost-always-right completion signal can survive on "almost always." Exit codes were right often enough that entire toolchains, CI systems, and deployment pipelines were built on trusting them, and the exceptions were rare enough to be absorbed as folklore (always check your pipes) rather than fixed as defaults. It took the accumulated weight of decades of silent 404s written faithfully to disk before the standard itself conceded that the default signal lies.
LLM pipelines are three years old, and their equivalent signal (an artifact landed, and it reads well) is wrong more often than exit codes ever were, in ways that are harder to see and attached to work that costs real money per invocation. We don't have fifty years, and we don't need them: the shape of the mistake is already fully documented, one level down the stack, with the fix's publication date in the standard for June 2024. The lesson transfers whole. A clean exit never meant the work was done. A present, plausible artifact doesn't either. Completion isn't a property your pipeline can read off its own outputs, it's a fact about the world, and the only checks worth trusting are the ones that go look.
false | tee report.txt exits 0 and leaves a zero-byte report.txt; with set -o pipefail the same pipeline exits 1. The default is specified behavior: a pipeline's exit status is its last command's, per the POSIX Shell Command Language.set -o pipefail into the standard, after decades as a ksh/bash extension (see also shellcheck #2555).Completion is a fact about the world. Keep the record that proves it.
Every rung on the ladder is a proxy except the one that goes and looks, and the check that goes and looks is only trustworthy if you kept an honest, tamper-evident record of what the worker actually did, not just what it claimed. The agent trust stack is that record made installable: chain-of-consciousness for the provenance of what an agent did step by step, plus ratings and verification so a downstream gate can check the work against ground truth instead of trusting a fluent artifact.
Read the Theory of Agent Trust · See Hosted Chain of Consciousness
pip install chain-of-consciousness · npm install chain-of-consciousness
Or the full stack: pip install agent-trust-stack / npm install agent-trust-stack