In 2018 an Uber test car struck and killed Elaine Herzberg in Tempe. The complete NTSB record shows every design decision behind the crash had an author on file, and criminal charges still reached only the person in the driver's seat. What that means for anyone who runs a human in the loop.
At 9:58 p.m. on March 18, 2018, a Volvo XC90 was driving itself north on North Mill Avenue in Tempe, Arizona. It was an Uber Advanced Technologies Group test car running the group's developmental self-driving system, and 5.6 seconds before impact it detected something in the road.
The system first classified the thing as a vehicle, and at other moments as an unknown object and as a bicyclist. Under its rules, a new label meant the system stopped using the object's tracking history, so every time it changed its mind it lost track of where the thing had been. It never worked out where the thing was going. It was Elaine Herzberg, 49, walking a bicycle across the road.
At 1.2 seconds out, the system concluded that a collision was imminent. It did not brake hard, because it had been built not to. The design "precluded activation of emergency braking for collision mitigation, relying instead on the operator's intervention." The operator in the driver's seat had spent nearly a third of the trip looking down toward the center console, where she had put her phone. She looked up about a second before impact and started to steer 0.02 seconds before the car hit Herzberg at 39 miles an hour. Herzberg died.
Twenty months later, the National Transportation Safety Board adopted its report on the crash. Two of its sentences matter here. The first gives the probable cause: "the failure of the vehicle operator to monitor the driving environment and the operation of the automated driving system because she was visually distracted throughout the trip by her personal cell phone." The second says what contributed: the Uber group's "inadequate safety risk assessment procedures," its "ineffective oversight of vehicle operators," and its "lack of adequate mechanisms for addressing operators' automation complacency," all of which the board traced to "its inadequate safety culture."
Both sentences are true, and they sit side by side in the report. Criminal charges followed only the first.
In 2004 the philosopher Andreas Matthias gave this kind of problem its first name. Machines that learn, he argued, leave the people who build and run them unable to predict what the machines will do, and the usual way of assigning responsibility breaks down with that. He called the space that opens up a responsibility gap.
In March 2019 the researcher Madeleine Clare Elish, then at Data & Society, named what fills it. Her paper opens with the Tempe crash, then builds its idea from Three Mile Island and Air France Flight 447. In a complex automated system, she wrote, a person can "become simply a component," accidentally or intentionally, one "that bears the brunt of the moral and legal responsibilities when the overall system malfunctions." She went on: "While the crumple zone in a car is meant to protect the human driver, the moral crumple zone protects the integrity of the technological system, at the expense of the nearest human operator."
Elish was working from the NTSB's preliminary report. Her paper appeared the same month prosecutors cleared the company, eight months before the final report. The full record, and the legal process that followed, show something her paper could not. The gap in Tempe was not the one Matthias described. The problem was never that nobody could know who decided what. The authorship of the crash was written down, and the law still reached for the person in the seat.
Start with the part that looks most like Matthias's gap, a perception system that changed its mind about what it saw. The NTSB traced even that to design. The system "never classified her as a pedestrian" because she was crossing where there was no crosswalk, "and the system design did not include consideration for jaywalking pedestrians." The rule that dropped an object's history when its label changed had an author too.
The rest of the report reads like a ledger, and most entries carry a date or a reason.
The report also counted the pedestrian's crossing outside a crosswalk while impaired, and Arizona's insufficient oversight of testing, as contributing factors. And it found that had the operator been attentive, "she would likely have had sufficient time to detect and react to the crossing pedestrian to avoid the crash or mitigate the impact."
What the report does not contain is a name. It calls the woman in the driver's seat "the vehicle operator" throughout, and it attributes every design decision to the company. That is deliberate, and the report says so on its opening pages: "The NTSB does not assign fault or blame for an accident or incident." The one process that wrote down every decision was built not to blame anyone for them.
The press did name the operator, and her name is easy to find. I have left it out. The public story of this crash attached one living person's name to the fault, and printing it again would repeat the pattern.
In May 2018 the preliminary report was already public. It said the car was set up not to brake hard under computer control and relied on the human to intervene.
In March 2019 the Yavapai County Attorney's Office, which took the case over from Maricopa County because of a potential conflict of interest, wrote that "there is no basis for criminal liability for the Uber corporation arising from this matter." The same letter recommended expert analysis of the video, to establish what the driver "would or should have seen that night."
In November 2019 the NTSB adopted its final report.
In 2020 the operator was charged with negligent homicide. Her lawyers said she had been looking at a messaging program Uber employees used, on a work phone. The NTSB relied on streaming records that showed her account playing a television show from 9:16 to 9:59 p.m. In July 2023 she pleaded guilty to endangerment, an undesignated felony that can become a misdemeanor if she completes probation. The sentence was three years of supervised probation, and the plea required restitution "to all victims," including insurance companies. Uber had settled with Herzberg's family less than two weeks after the crash.
The company's criminal question was closed before the full record existed. Hers stayed open for four more years. She was at fault, and the NTSB said so. She also took the fall for everything in that second sentence.
None of this needs a cover-up to explain it. The mechanism is plainer.
Criminal negligence, as it is usually applied, looks for a person, an act and a moment. A video of a face turned down toward a phone fits that shape. A one-second braking delay does not. It was set in advance as a tradeoff between false alarms and missed hazards. It had many hands and no single moment, and its reason sounded responsible.
The evidence leaned the same way. The operator's trip is documented glance by glance from the inward camera and minute by minute from the streaming account. The decision to run one operator instead of two survives as a sentence of explanation. And the camera that recorded her belonged to the company. It was installed to watch the operators and, by the supervisors' own account, rarely watched. An oversight tool the company barely used for oversight became the record of her failure.
This is what the full record adds to Elish. The crumple zone in Tempe was not built from ignorance. It was built from timing, because the company's criminal exposure was settled before the investigation finished, and from legibility, because our institutions read an act far more easily than a decision. Neither requires anyone to lie, which is why the pattern repeats.
None of this makes the phone acceptable. She was hired to watch the road, and the phone policy was covered in her first week of training. But the job had a known flaw, and the flaw had been in print for 35 years.
In 1983 the psychologist Lisanne Bainbridge published "Ironies of Automation," about process control rooms and flight decks. "A more serious irony," she wrote, "is that the automatic control system has been put in because it can do the job better than the operator, but yet the operator is being asked to monitor that it is working effectively." She cited the vigilance research as well: "it is impossible for even a highly motivated human being to maintain effective visual attention towards a source of information on which very little happens, for more than about half an hour."
The Tempe trip included 39 minutes on a public road. The NTSB made Bainbridge's point in its own words: on the human capacity to monitor an automated system for its failures, "research findings are consistent," and "humans are very poor at this task." It called the operator's distraction "a typical effect of automation complacency." A design that makes a typical human failure the last line of defense has already decided who will be standing there when it fails.
Software teams run a smaller version of this experiment all the time, and their usual answer is different.
On February 28, 2017, an Amazon S3 team member "using an established playbook executed a command" meant to remove a small number of servers. "Unfortunately, one of the inputs to the command was entered incorrectly." Too many servers came out, and while its subsystems restarted, "S3 was unable to service requests" in Amazon's Northern Virginia region. AWS's public summary names no one. It says "the tool used allowed too much capacity to be removed too quickly," and "We have modified this tool to remove capacity more slowly and added safeguards." Google's SRE book states the principle: "You can't 'fix' people, but you can fix systems and processes to better support people making the right choices."
Uber itself answered Tempe the same way. After the crash the company removed action suppression, taught its system that pedestrians cross midblock, kept an object's history when its label changed, made the Volvo's collision warning and automatic braking available during testing, went back to two operators when testing resumed in December 2018, and installed a monitor that chimes when the driver looks away and sends a report to a supervisor. Every fix was a fix to the system. Together they are a map of where the company located the causes. The charges followed a different map.
You probably do not write braking logic. You almost certainly run a human in a loop: a reviewer approving an AI agent's pull requests, an on-call engineer confirming a deploy, an analyst clearing fraud flags. The EU AI Act, whose human-oversight rules apply to stand-alone high-risk systems from December 2027, requires those systems to be designed so they "can be effectively overseen by natural persons," with the people doing the overseeing able to stay aware of "the possible tendency of automatically relying or over-relying on the output," which the Act calls automation bias. Tempe shows what that requirement looks like when it is met only on paper.
Four habits follow from the record.
Give the human a job a human can do, and measure it. If the person's role is to catch the machine's rare failures, you have designed a vigilance task, and Bainbridge put its limit at about half an hour. Measure how long the human gets to react, whether anything tells them when it matters, and how often they are asked. Tempe's design gave the operator a second of silence, and a warning only once its own braking limits could no longer prevent the crash. A step that asks one reviewer to approve everything an agent produces all day is a signature, not a check.
Treat a muted alarm as a safety decision with an owner. Action suppression existed to quiet false alarms, the same reason on-call teams mute noisy pagers. Muting is sometimes right. Record who muted it, what it no longer catches and who covers the gap, and tell the person who has just become the backstop.
Log decisions as carefully as you log operators. If your system records every click the human makes, it should also record who removed the second reviewer, who disabled the check that kept failing in CI, and why. This is not about blame. A blameless review still has to know which decision to ask about, and a log that only sees the operator will end every story at the operator.
Use your monitoring before you need it as evidence. Uber had cameras on its operators and rarely looked. Review the human side of the loop while it can still change the outcome.
The responsibility gap has a name now. What Tempe adds is how ordinary its construction was. Every item in that ledger, from the one-second hold on braking to the empty passenger seat, was somebody's decision. None of them was hers.
The Tempe record kept the operator glance by glance and kept each design decision as a sentence of explanation. If your system puts a person in the loop, log the decisions as carefully as the person: Chain of Consciousness keeps a tamper-evident record of who changed what in an agent's setup and why, such as who removed a reviewer or muted an alarm, so a review can find the decision and not only the person standing at the end of it.
pip install chain-of-consciousness npm install chain-of-consciousness
Or start without installing anything: Hosted Chain of Consciousness.