GitLab counted five backup and replication techniques in 2017. None of them worked, and six of my eight green lights would have looked identical.
On the night of January 31, 2017, a GitLab engineer trying to fix database replication deleted a directory on the wrong server. About 300GB of production data shrank to 4.5GB before the command was stopped. That part of the story is famous. The part that matters here is what happened next: GitLab went to restore from backup and discovered, live, in front of everyone, that they effectively had none. Their working notes, written during the outage and quoted by TechCrunch the next morning, contain one of the great sentences of operations writing: "out of 5 backup/replication techniques deployed none are working reliably or set up in the first place."
Five, by GitLab's own count of backup and replication techniques, and every one was a memorial. The nightly pg_dump job had been writing to S3 for months while the bucket sat empty: the database had been upgraded to PostgreSQL 9.6, the backup tool was still the 9.2 binary, and when pg_dump meets a version it does not match, it refuses to run. The cron emails that would have said so were silently dropped because they lacked the DMARC authentication GitLab's own mail policy required. The green light was on. The channel that could have said no was dead. Nothing behind the light had worked for a long time, and nothing about the light showed it. They lost about six hours of data, roughly 5,000 projects, 5,000 comments, and 700 new user accounts, and what saved them from losing far more was not any of those five. It was a snapshot an engineer had taken by hand, six hours earlier, for an unrelated test.
I operate a fleet of automated agents, and one afternoon this week the fleet handed me the same lesson four times in miniature. I wrote recently about one way checks lie: the empty result, where "found nothing" and "looked at nothing" print the same bytes. This piece is the other face of that coin. Not the clean zero but the stale success. When I lined up every success signal my fleet trusts and asked one question of each, six of the eight were commemorative, and four of them burned us inside one working day.
I started calling these memorials, to keep them apart from receipts. A receipt is generated by the transaction. A memorial commemorates something that once occurred. Both look like confirmation. Only the receipt is evidence about what you just did, and the test that separates them is a single question:
If the action had silently done nothing, would this signal look any different?
That evening I played the question against the signals we trust, and kept score:
| signal | would it look the same if nothing happened? |
|---|---|
| "Cover uploaded successfully!" banner on a title that already has a cover | yes. It is a memorial, not a receipt |
| The "No Cover Uploaded" placeholder is gone | yes, if it was never there to begin with |
| The upload call returned without raising | no. It returned an error string instead, which we discarded |
| "Done. 1 sent." from our mailer | no. That line comes from the SMTP round trip itself |
| A pre-send audit stamp reading PASS | yes, if the draft changed after the stamp and nobody re-ran it |
| A safety hold described as "armed" in a code comment | yes. The comment is prose; the lookup table is the arming |
| A test that "was fixed" according to its commit message | yes, until you run it |
| A test that passes when you do run it, because its assertion cannot fail | yes. Green either way, run or unrun, right or wrong |
Two of eight are receipts. The other six would read identically in a world where the action failed. GitLab's five techniques all sat in the bottom six's territory: a backup job that "ran" nightly, a snapshot schedule that "existed," an alert channel that "would" fire. The ratio is the point, and it was the point in 2017 too. Most of what both fleets accepted as confirmation was commemorative.
The eighth row deserves its own sentence, because it is the hardest case and it was found the same day, by a reviewer checking this essay's own claims against our systems. A shared test suite asserted rank progression in a game engine by checking that the word for the next rank appeared in a status line. The status line names the rank you hold and the next gate: "rank APPRENTICE, next gate: journeyman." So the check for "journeyman" passed one rank early, every time, across three separate crafts. Row seven is a memorial until you run it. Row eight is the harder thing: a memorial that survives being run. The test ran. It passed. Running it told you nothing, because it was green in both worlds.
The commit message said "fixed." The fix added a parameter to a hold-alert function so a silent failure would page us instead of passing quietly. The unit test proving the fix was in the repository, named for it, referenced by it. It had also been failing for 31 consecutive cycles, because the fix's own call site never passed the new parameter. Nobody noticed, because nobody was running the suite that contained it; the commit message had already told us the story we wanted, and a message is a memorial by construction. It describes an intention at write time. It does not update when the world disagrees.
The discarded error string from our cover upload is the same shape one layer down. The call ran, and it answered, and the answer was no. The tool threw the answer away and went looking for a signal it liked better: the success banner on the page, which had been sitting there since the original upload weeks earlier and was going to say "Cover uploaded successfully!" in every possible world. If your process runs the real check and then consults a memorial anyway, you have built a machine for overruling receipts.
The question to ask: can you point to output this specific run produced? Not output that exists. Output that would not exist if the run had not happened. GitLab's empty S3 bucket is the canonical negative: the job "ran" nightly and produced nothing, and no one asked the bucket.
Our dispatch queue reported its status as "draining." Draining is a comforting word: work is leaving, the system is alive, come back later. Meanwhile the head of one lane was a job waiting on a browser session that nothing was going to release, and the lane sat blocked for 1.4 hours while the status said draining the whole time. The status was computed from queue arithmetic that had no input capable of representing "an idle browser is holding this." It was not lying, exactly. It was answering a smaller question than the one we heard, and it had no vocabulary for the failure actually happening.
That is the practical meaning of "can it say no": not whether a failure state exists somewhere in the enum, but whether the failure you care about has a path to the output. GitLab's version was crueler: the failure had a path, and the path was dead. The cron job did try to complain, nightly, by email, and the mail system dropped every one for a missing authentication stamp. A signal that cannot express your failure, and a signal whose failure messages never arrive, converge on the same terminal state: success reported through disaster, with complete sincerity.
The cannot-fail assertion is this question in its purest form. The probe is cheap: cause the failure once, on purpose, and watch the signal. We now keep a negative-control switch we run against that test suite, which puts the fixed assertion against the rank below, and the suite is only trusted while that control still fails. It is a switch we run against the suite, not a check inside every pass, and the difference matters in a piece about signals that are not what they look like.
The outreach letters passed every gate we had: compliance language, formatting, suppression, spacing. Then we compared the honorifics in the letters against the records the letters were built from, which no gate did, and found that 94 of 107 honorific tokens had been inferred from first names rather than read from the record. Someone named Leslie gets a guessed title; so does Jordan, and a person from a culture whose given names the inference has never seen. Every gate was green because every gate examined a property of the letter, and the defect was a relation between the letter and the record. No amount of staring at an artifact detects a disagreement with a source you never open.
This is the failure shape that survives longest, because every gate involved is genuinely working and genuinely catching other defects. The system of gates still never looks at the claim. When a check goes green, the question is not whether the check is healthy. It is whether the delta you intended to put into the world is inside any check's field of view. Ours was not, 94 times out of 107.
Strip the three questions down and they converge on one procedure, which is the only confirmation I have found that cannot quietly become a memorial.
Read the state before you act. Act. Read the state again, through the same instrument. The confirmation is the difference between the two reads, and nothing else. If the reads are equal, nothing happened, whatever the page says, whatever the banner celebrates, whatever the commit message promised. If they differ in the way you intended, you have a receipt, and you manufactured it yourself out of two observations no stale artifact can fake.
Notice that this is exactly what saved GitLab. Not a status page, not a schedule, not a policy. A snapshot taken by hand: a read of the state that verifiably existed, sitting where a restore could diff the world against it. Every one of the five techniques GitLab counted reported existence. Only the manual snapshot could demonstrate a difference.
Our cover tool got the same treatment the day after it burned us. It now fingerprints the cover image before the upload and after, and reports the pair. When the fingerprints match, it prints "nothing changed," which is a sentence the success banner cannot pronounce. The first time it ran, that is what it printed, because the underlying error from that afternoon was still there. The banner on the page still said "Cover uploaded successfully!" It always will. We just stopped asking it.
Before, act, after, diff. It costs one extra read. Every signal in the eight-row table that fails the question fails it by skipping a read: the banner has no before, the commit message has no after, the audit stamp froze its after while the world moved on, the cannot-fail assertion diffs against nothing at all. Two boring reads are the entire difference between measuring your action and commemorating someone else's.
Sources and provenance. The GitLab figures and the quoted sentence are from GitLab's public incident notes (the "problems encountered" list, written during the 2017-01-31 outage) as quoted by TechCrunch on 2017-02-01, and from GitLab's official postmortem, "Postmortem of database outage of January 31" (about.gitlab.com, 2017-02-10): about six hours of data, roughly 5,000 projects, 5,000 comments, 700 new user accounts, pg_dump 9.2 against PostgreSQL 9.6, cron failure mail dropped for missing DMARC authentication. The four fleet figures (31 cycles, 1.4 hours, 94 of 107, and the discarded error string) were re-read on 2026-08-24, before this revision shipped, from the day's operating journal and from the outreach method record that logged the honorific audit (54 drafts, 107 tokens, 94 not in the record, measured over 267 outreach runs). The first draft of this essay said those numbers had not been recounted. A reviewer pointed out that an essay about stale confirmations should not itself ship inherited numbers, and that criticism was correct, so now they are receipts.