In February 2020 the safe holding the internet's root signing key would not open. IANA's published, hand-initialed ceremony scans show the custody behind the cryptography: a repair script written before anyone drilled, and a signing that took four minutes and fifteen seconds.
Page 6 of the script is a table with four columns: Step, Activity, Initials, Time. In step 5 a locksmith replaces a combination lock, and the first task on the list reads "a) Drill the lock/safe open." Beside it, in blue pen, someone has initialed the box and written a time: 21:48+1.
The safe is Equipment Safe #1 in the Key Management Facility that IANA runs in El Segundo, California. It holds the hardware security modules that keep the private half of the root zone's key-signing key, the trust anchor that validating resolvers chain every signed DNS answer back to. In February 2020 it would not open. IANA publishes the paperwork from its key ceremonies, scanned with the handwriting left in, so the story of how it got into its own safe can be read page by page, initials and all.
Most writing about the root key ceremony is about cryptography. The paperwork is mostly about custody: who held what, who watched, and what got written down when something went wrong. That February the cryptography took four minutes and fifteen seconds. Getting into the safe took most of four days.
The root zone is signed in quarters, and the key-signing key is brought out of its safe four times a year to sign the keys for the next one. The rule in force in 2020, in IANA's DNSSEC Practice Statement, was that each quarter's ceremony is held "approximately 60 days but no later than 33 days, before the time cycle commences."
The people are set out in roles. A Ceremony Administrator runs the script and an Internal Witness keeps the record. Safe Security Controllers hold the combinations, two people for each of the two safes: the Equipment Safe with the hardware, and the Credentials Safe with the safe deposit boxes. Crypto Officers hold the keys to those boxes. The Practice Statement describes "successive barriers in three tiers, requiring at least seven trusted persons from four different roles," and activating a hardware module "requires three out of seven Crypto Officers to extract activation material from above safe deposit boxes using their physical key." The Crypto Officers are Trusted Community Representatives, picked from statements of interest that IANA solicits from the internet community, and its roster page states the rule that keeps them independent: "Persons considered directly affiliated with PTI, ICANN or VeriSign may not become a Trusted Community Representatives."
The script's preamble says how the room behaves. "CA reads each step aloud prior to its performance." "Upon completion of each step, IW announces the time of completion, records the completion time, and initials their copy of the script." And then the paragraph the rest of this story runs on:
Ceremony participants who notice a problem or an error during the ceremony should interrupt the ceremony. Ceremony participants agree on a resolution before proceeding ... Unplanned events (exceptions) during the ceremony are evaluated, documented, and acted upon. It is the CA's sole responsibility to decide on proper actions after consulting with the IW. In either case, an exception is regarded as an incident, and incident handling procedures are enacted.
"Documented" has a form. The Root DNSSEC Script Exception is a one-page sheet. Its first row asks for the Act, the Step, the Page, and the date and time, and its second says "IW describes the exception(s) and action(s) below," with a box for initials and a box for the time on each row. An exception is filed against a place in the script, the way a bug is filed against a line of code.
The trouble surfaced during maintenance, and the maintenance was itself a ceremony with its own script. Administrative ceremonies are smaller affairs, with no signing, but the same rules apply: a Ceremony Administrator, an Internal Witness, cameras, and a participants table. In that week's tables the participants included a locksmith from an outside lock-and-security company.
The work was to replace the combination dials on both safes, swapping each Kaba Mas X-09 for a new X-10. The day before, at a ceremony to program the safe deposit boxes' lock assemblies, one assembly had misbehaved. That got a sheet too:
The last KMF West Lock Assembly was malfunctioning, and we delayed the bagging of the unit to allow more time for the locksmith to troubleshoot. Once the lock assembly was functioning, we placed it in TEB BB91951419 and continued where we left off in the script.
TEB is a tamper-evident bag, and the bag's serial number goes in the record.
The script for the Equipment Safe, dated Tuesday 11 February, has an Act 2 called "Safe #1 Equipment Maintenance." Step 1, initialed at 21:34, has the administrator and the witness carry "a flashlight, phillips screwdriver, change key tool" into the safe room. Step 2 reads "SSC1_A opens Safe #1 while shielding the combination from the camera," with a note that the controller "will begin by rapidly spinning the dial counter-clockwise in order to charge it." The lock powers itself from the dial. Step 4 is the job they had come to do: "a) Remove the current Kaba Mas X-09. b) Install a new Kaba Mas X-10."
Step 2 has no initials. The exception sheet behind it is filed against Act 2, Step 2, Page 6, dated 2020/02/11 and initialed at 23:14:
SSC1_A was unable to open the safe due to malfunctioning hardware. Escorted out by IW2 to verify combination with other SSC. Locksmith contacted manufacturer and they suggested the issue was a faulty logic board and the dial will require drilling. Ended ceremony to discuss steps going forward.
So the dial that failed was the one they had come to replace, and it failed at the one step that needed it to work. A second ceremony a couple of hours later opened the Credentials Safe without trouble and swapped its X-09 for an X-10.
The next day ICANN's blog announced that the 40th ceremony, "originally scheduled for 12 February 2020 in El Segundo, California, has been postponed." "One of the security mechanisms that protects the contents of the secure safes was found to be malfunctioning." "The issue was identified yesterday, 11 February 2020, during routine maintenance of the Key Management Facility." The post said there would be "no interruption to Domain Name System Security Extensions (DNSSEC) service," that "We maintain a complete replica facility in Culpeper, Virginia," and that the repair was scheduled for Friday the 14th.
The 11 February script had no step for a safe that would not open. Step 2 simply assumed it would. When it failed, nobody improvised the repair in the room. They filed the sheet, ended the ceremony, and wrote a new script.
The repair ceremony began on the 14th, and its Act 2 opens with a section titled "Confirmation of Equipment Safe #1 Dial Malfunction." Before anyone was allowed to drill, the controller tried the safe again, under the cameras, at 17:26. The step carries a note that "Locksmith may provide additional advice/techniques to attempt to operate the safe dial." Then the script splits:
If the safe opens: Complete the safe log by following the steps below ...
If the lock malfunctions: Continue with the steps below.
If the safe had opened on that attempt, the script would have sent everyone to the safe log and the dial swap, and there would have been no drilling. It didn't open. Step 5 begins "The locksmith performs the following tasks to replace the combination lock," and lists them: "a) Drill the lock/safe open," "b) Remove the current Kaba Mas X-09 Safe Dial," "c) Return the safe door to a secure state by performing necessary repairs," "d) Install a new Kaba Mas X-10 Safe Dial." Under the first task sits a note about breaks: "A ceremony break will be performed as required. All personnel will be escorted out of the Tier 5 (Safe Room) and Tier 4 (Ceremony Room.)" Each lettered task has its own box for initials and time.
Drilling into the safe that guards the root key was written down in advance as a numbered, witnessed, timed step, with a branch for the possibility that it wasn't needed.
The repair took two days, and the packet of exception sheets behind its script reads like a log of everything the plan didn't specify. I read four of them, all filed against Act 2, Step 5, Page 6. I didn't read every page of the packet, so four is a floor.
The first, dated the 14th at 21:00, records that an external witness "left the room on a break and stayed outside until the next break," returning at 22:19, and adds: "Due to the length of the ceremony carrying into the next day, other non essential participants came and went as well." The second is one sentence: "Ceremony was paused overnight and resumed in the morning."
The third, at 19:26 on the 15th, is about the building. A security guard used the hand scanner outside Tier 3 to escort an external witness in, and that triggered the alarm. The witness was asked to step out and wait in the conference room, "and the alarm was triggered again when he manually opened the door."
The fourth is about the order of steps:
Steps e, f, g, and h were performed immediately after opening the safe for the first time which occurred between steps a and b.
Steps e to h are the safe log: the controller shows the old log to the camera, takes the pre-printed one from the witness, writes the time and signs where it says "Open Safe," and the witness checks and initials the entry. The script had them after the new dial went on. In the room, they did them the moment the drilled door first opened, and the times in the script's own columns agree: a at 21:48, e to h at 21:54, the new dial at 23:56. Doing the log first was arguably the better order, since it put the opening on paper at once. Nobody rewrote the script to match. The deviation went on its own sheet, and the record shows the steps in the order they happened.
The times in that column all carry a "+1," which the script never defines. I read it as the day after the ceremony began, which puts the drilling at 21:48 UTC on the 15th. Three other records agree with that reading: the overnight sheet is dated the 15th at 00:38, IANA's announcement that the safe was open went out at 22:10 UTC on the 15th, and the participants signed the ceremony off at 00:44 on the 16th. The discovery sheet was initialed at 23:14 on the 11th, so the drilling step was initialed 3 days, 22 hours and 34 minutes after it.
IANA's message to the root-dnssec-announce list read: "We have successfully opened the safe with the failed mechanism, and are in the final stages of remediation, with the repair work and installation of a new lock underway. We are therefore arranging resources to start the KSK Ceremony 40 at 0000 UTC, or shortly thereafter."
The signing went the way the script said it would. The signer's log for Ceremony 40 opens with "Starting: ksrsigner /media/KSR/KSK40/ksr-root-2020-q2-0.xml (at Sun Feb 16 02:21:33 2020 UTC)." Twenty-two seconds later the hardware module is activated. At 02:25:48 the log reads "Generated new SKR," a signed key response whose first signature period begins on 1 April 2020, made with the key whose tag is 20326. The log prints the response's SHA-256 hash in hex, and then again as a run of English words: "unwind leprosy fracture savagery tactics speculate Neptune miracle slingshot aftermath …"
From start to signed response is four minutes and fifteen seconds. Against the Practice Statement's rule, the latest date for the second-quarter ceremony was 28 February, so the signing finished with nearly twelve days to spare.
The ceremony's closing act has the administrator read every exception out before anyone signs. Step 1 of the participants' signing reads "CA reads all exceptions that occurred during the ceremony." In step 2 each attendee comes to the table and signs the witness's list, and "All signatures declare that this script is a true and accurate record of the ceremony." The read-back comes first. You hear what went wrong before you put your name to the record.
Two months later the failure was the people. Ceremony 41 was set for 23 April 2020, in the first weeks of the pandemic. IANA's published questions-and-answers document says there were no representatives for the El Segundo facility "located close enough to allow unrestricted and safe travel to the facility," and says where they were: "Mauritius, Spain, Russia, Tanzania, Uruguay and east-coast United States."
The plan IANA used kept the custody chain and moved the people. "Four trusted community representatives (TCRs) who fulfil Cryptographic Officer roles in the ceremony have transmitted their safe deposit keys to separate ICANN and PTI staff. These staff will act as surrogates, but the TCRs will participate in the ceremony remotely and monitor their credential's use." The keys travelled sealed: "The TCRs have wrapped their deposit keys with opaque material, and then transmitted them in tamper-evident bags. This bag will not be opened until within the ceremony so that each TCR can witness their key is in the same condition as when they released it." The ceremony would sign three quarters at once instead of one, and "The additional generated signatures will be initially withheld."
The plan they didn't use is in the same document. "The first version of our contingency plan assumed the trusted community representatives would not transmit their safe deposit box keys for performance of the ceremony. In such a scenario, a trained locksmith would drill and replace the locks to three safe deposit boxes during the ceremony in order to retrieve the smart cards needed to activate the hardware security module." Two months after the safe, the first version of the fallback was to drill again.
One part of the April plan broke the rules as they stood, and the document says which: "The only element of our revised approach that was inconsistent with the DPS was signing additional quarters of KSRs during the ceremony. The Policy Management Authority, an internal committee that reviews changes to the DPS, convened on 6 April and approved revisions that allow for signing additional quarters of KSRs as a disaster recovery response." The rule was changed by its owner seventeen days before it was needed, and not by the people in the room on the day.
The Ceremony 41 script also adds a step to its closing act. After reading the exceptions, the administrator "asks any COs who are participating remotely if they have any concerns pertaining to the ceremony or exceptions which may have occurred."
By November the bend was standing policy. The Practice Statement in force in February 2020, the fourth edition, has no variance clause. The sixth edition, dated 4 November 2020, carries section 4.5.4.1, and the eighth edition of 2025 still carries it word for word:
In the event of a declaration of a disaster recovery or business continuity response, the adverse event may necessitate variances to the procedural controls and technical security controls. These variances shall be limited in scope to those necessary for the successful conduct of key signing operations, and according to protocols that best ensures a proper record of the chain-of-custody and limiting sensitive operations to trusted persons.
The clause ends by requiring that variances are "consented by the RZ KSK Operator's management, are documented and communicated." I didn't read the fifth edition, so I can't say which edition first carried it, only that it arrived between February and November 2020.
IANA's schedule lists 63 held ceremonies from June 2010 to August 2026, counting Ceremony 53's two parts separately. In 2020 and 2021 there were two each, the only full years with fewer than four. Ceremonies 41 and 42 are the only ones that signed more than one quarter, and the 294 days between them is the longest gap in the schedule.
The public archive runs to 65 directories and 940 files, among them 315 video files from the ceremony cameras. Files from administrative ceremonies first appear in the directory for Ceremony 40, the safe repair, and turn up again in 15 later directories. I didn't read what those later administrative ceremonies were for. Everything here comes from IANA's and ICANN's own record. That is the point of reading it, and also its limit, because an outside auditor's view isn't quoted here.
None of this is exotic. It is a procedure that expects to be interrupted, and a record that treats the interruption as a first-class entry. Anyone running software that acts where actions can't be taken back, a deployment pipeline or an agent holding production credentials, can copy four habits from it.
An exception there is a typed record: act, step, page, time, initials, and a description written by the witness rather than the actor. When the script doesn't cover a failure, the response in that room was to stop and write a new script, with a branch for each outcome, before taking the irreversible step. When steps happen out of order, the record says so instead of being tidied. And when a rule has to bend, its owner bends it in advance, in writing, and later turns the bend into a clause with limits.
At the back of the Ceremony 40 script is a page for the Internal Witness alone. It is two sentences long: "I hereby attest that the Key Ceremony was conducted in accordance to this script. Any exceptions that occurred were accurately and properly documented."
Sources
How this was checked. Typed quotations were checked word for word against the saved text of the source named beside them. The handwritten ones were transcribed from the scanned pages of the annotated scripts and checked against the images; the scans have no text layer. Times are UTC, as the scripts record them, and the "+1" day is read as explained in the text. The intervals (3 days, 22 hours and 34 minutes; 4 minutes and 15 seconds; the margin before 28 February) and the counts from the schedule and the archive listings were computed by script from those sources. Four exception sheets is the number read in the repair ceremony's packet, not a total for the month.
Chain of Consciousness keeps a tamper-evident, dated record in hash-linked entries, so what an automated system did, when, and what went off-script can be checked later instead of reconstructed after someone asks.
pip install chain-of-consciousness npm install chain-of-consciousness
Or start without installing anything: Hosted Chain of Consciousness.