One flight plan stopped the UK's flight planning system in under 20 seconds. The CAA panel's timeline shows the next six hours: a four-hour buffer of stored plans, an on-call ladder with its expert at home on a bank holiday, and the manufacturer, called last, which estimates it had a solution within half an hour.
"12:32 · NERL · 04:00 · Stored flight plan data exhausted."
"12:39 · NERL · 04:07 · Teams call with level 3 engineer and software supplier, Frequentis Comsoft"
Those two lines sit near the end of a timeline in CAP 2993, the final report of the independent panel the UK Civil Aviation Authority set up after NATS's flight planning system failed on Monday 28 August 2023. NERL is NATS (En Route) plc, the UK's main provider of air traffic services. The third column counts the time since 08:32, when a single flight plan stopped the system. At four hours, the store of flight data that let controllers carry on without the system ran out. Seven minutes later, for the first time that day, NATS's engineers were on a call with the company that wrote the software.
It was the summer bank holiday (NATS report, p. 17), and airlines and airports told the panel the day was "one of the busiest days of the year in terms of passenger numbers" (¶2.31). NATS counted more than 1,500 cancelled flights that Monday (NATS report, p. 4). The panel puts the passengers affected by cancellations and delays at "over 700,000" (¶3.1), the cost to airlines at "approximately £65m", and the total cost at "in the region of £75m to £100m" (¶5).
The software stopped in less than 20 seconds, as safety critical software is designed to. The panel's report records the six hours that followed nearly minute by minute. Paragraph and figure numbers below are the panel's (CAP 2993); "NATS report" is the company's own preliminary report on the incident.
The flight was BF371 (Appendix D). Its airline had filed "an ICAO4444 compliant flight plan" with Eurocontrol's flight planning system, IFPS, which accepted it, and the aircraft was cleared to depart at 04:00 (NATS report, p. 9). IFPS passes a plan on to NATS four hours before the aircraft reaches the UK's airspace, so at 08:32 the plan arrived at FPRSA-R, the NATS system that extracts the UK portion of each plan for the computer that shows it to controllers (NATS report, p. 9).
In its converted form the plan listed two waypoints, far apart and both outside UK airspace, with the same name (NATS report, p. 9). Duplicate names are not an error in themselves. ICAO and others have worked to remove them, but duplicates exist around the world, and the current standards say "such identical designators should be geographically widely spaced" (NATS report, p. 9).
The panel walks through what the software did next (¶2.7). It found the entry point into UK airspace, APSOV. Searching for the exit point, it tried SITET and then ETRAT, and rejected both because neither was in the original flight plan. The third candidate was DVL. DVL was in the original plan, so the software accepted it as the exit. "The code for Deauville is DVL," the report says, and Deauville is in France. The DVL in the airline's plan "referred to Devil's Lake in North Dakota and not to Deauville in France" (¶2.7).
That left a flight whose exit from UK airspace came before its entry. "Recognising this as being not credible," the primary system raised a critical exception, disconnected itself and went into maintenance mode (¶2.8). The backup system then "applied the same logic to the flight plan with the same result" and did the same (NATS report, p. 10). From the plan's arrival to both systems sitting in maintenance mode "took less than 20 seconds" (NATS report, p. 11). Since it went into service in October 2018, FPRSA-R had processed more than 15 million flight plans without losing both systems at once (NATS report, p. 7).
NATS's report defends the stop. "Safety critical software systems are designed to always fail safely," it says. A system that cannot continue in a demonstrably safe way moves into a state that needs a person (NATS report, p. 10). In the same passage it allows that "Clearly a better way to handle this specific logic error would be for FPRSA-R to identify and remove the message and avoid a critical exception."
The panel adds two things the company's report did not. The duplicate waypoint case was not unforeseen. NERL's design document identified it, and Frequentis Comsoft, the software's manufacturer, told the panel that the coding turned out more complex than expected and "the logic was missed during the coding process" (¶2.34). The missing code, the panel writes, "would have prevented the events of the 28 August 2023 from ever happening" (¶2.34). And the system had no way to set one bad plan aside. Frequentis said that reflected the importance of never losing a flight plan message, but agreed, "with hindsight", that feeding the backup the same data that had stopped the primary should have been designed out (¶2.37). Both companies added that the backup's main purpose is to cover hardware failures in the primary and failures of its interfaces with other systems (¶2.37). The panel's conclusion is a sentence long: it "remains of the view that the system design should have provided better mitigation (e.g. quarantining) of data that would have caused the FPRSA-R system to cease operating" (¶2.40).
A system that stops safely still leaves the aircraft in the air, and NATS had a buffer for that. The national airspace system keeps "4 hours of previously filed flight data" so that operations can continue after automatic processing is lost, and after that, plans can be entered by hand (NATS report, p. 8). At 08:32, in the company's words, "the 4 hour buffer to manual flight plan input commenced" (NATS report, p. 11).
Manual entry is slow. At peak, FPRSA-R handles about 900 flight plans an hour, and manual input brings that down to about 60; there were seven terminals for it at Swanwick (¶2.14). The panel judged the manual fallback sound on safety, since "no safety occurrences were reported as a direct result of the event", and it credits NERL's staff for that. But at under a tenth of the normal rate, manual input could not carry the traffic without restrictions (¶2.15).
The restrictions came in steps. NERL's first formal contact with Eurocontrol was at 10:43, and from 11:00 all of UK airspace was limited to 360 flights an hour. At 12:20 the limit fell to 40 an hour for the airspace run from Swanwick and 20 for Prestwick's, and at 13:00 to 20 and 10 (¶2.17, ¶2.18).
NERL's engineering support has four levels, laid out in the panel's Figure 2.2. Level 1 engineers are on site around the clock, each competent on 50 to 60 of the most critical systems. Level 2 are system experts, "onsite during working days, rostered on call but offsite (within 1hr) outside of working hours". Level 3 are design experts, "not rostered but available by request". Level 4 is the equipment manufacturer.
Appendix D shows the day climbing it. The times are local, with the elapsed time in brackets:
Once the manufacturer was on the call, the end came quickly. Frequentis estimates that it identified a solution "within 30 minutes of being contacted", and it credits NERL's engineers with having already ruled out several causes (¶2.32). The fix was to move the offending plan out of the message switch's pending queue and into a pause queue. "Once this was made known to NERL's engineers, resolution of the fault was quickly achieved," the panel writes, and Frequentis told it that it would not have been reasonable to expect NERL's engineers to find the cause, or the fix, without its input (¶2.25). With the system back, the next four hours of flight plans went through "in approximately 9 minutes" (NATS report, p. 12). Restrictions began to ease at 15:24 and were all lifted by 18:03 (¶2.18).
The panel lists eight factors that made finding and fixing the failure "more protracted than it might otherwise have been" (¶2.28). The Level 2 engineer was on call rather than on site, and once remote options were exhausted, "it took 1.5 hours for the individual to arrive on-site to perform the necessary full system re-start which was not possible remotely". The team followed escalation protocols under which the Level 3 engineer's help was not sought "for more than 3 hours after the initial event". The Level 3 engineer did not know the fault message and needed Frequentis to interpret it. Frequentis, with "a unique level of knowledge of the AMS-UK and FPRSA-R interface", was not asked for more than four hours. NERL's joint decision-making model meant "no single post-holder with accountability for overall management of the incident". The state of the data in the message switch was not clearly understood, and there was "a lack of clear documentation identifying system connectivity". The last item is the smallest and the most familiar: "The password login details of the Level 2 engineer could not be readily verified due to the architecture of the system."
Behind the on-call item is one sentence in the next paragraph. "NERL's arrangements for the availability of engineering support are based primarily upon the level of engineering work planned (particularly maintenance activities) and not on the level of demand for air traffic services" (¶2.29). One Level 2 engineer is on site in normal working hours, when the planned work happens. "On weekends and public holidays, when specialist maintenance is not scheduled, Level 2 engineers are rostered to be available on-call at home" (¶2.29). On the bank holiday, with no maintenance planned, the roster had its expert at home, on a day the airlines rank among the busiest of the year. NERL's reply to the panel was that every day of the summer season is very busy, and that suspending routine work on public holidays is what allows the on-call arrangement (¶2.31).
The panel asked NERL what it would take to keep Level 2 engineers on site at Swanwick and Prestwick at all times. NERL calculated "an additional 102 engineering staff", at between £11.5M and £16.9M a year, and warned that engineers kept on site would not build their expertise the way the current ones do, on systems projects (¶2.29). It is not pursuing that option. Among the others under consideration are "enhancing remote access for critical systems", a reassessment of engineering service level agreements and more support from suppliers for critical systems (¶2.29).
The panel did not ask for round-the-clock cover either. Its view is that cover at all times "may not be justified on grounds of either safety or cost", but "there is a case for making such an enhancement during the busier summer period", and that the cost "should be seen in the context of the overall cost to the industry and to passengers of the incident on 28 August 2023" (¶2.30). Its recommendation R2 asks NERL to reconsider its "engineering resource management arrangements to provide timely onsite coverage with engineers of sufficient skill levels that are matched to aviation system demand" (¶2.30). The panel made 34 recommendations in all.
Two clocks started at 08:32. One was the buffer: four hours of stored plans, kept for the loss of automatic processing. The other was the time to reach the one party who could read the fault. That route ran through a ladder whose second rung was at home because no maintenance was scheduled, and whose top rung, the manufacturer, came last by design. On 28 August the second clock ran seven minutes past the first.
A team that keeps a fallback for a stopped system can measure the same two things: how long the fallback carries the load, and how long, on the thinnest day of the roster, it takes to get the person who understands the error onto a call. And it can check who its ladder calls last. NERL's protocol called the manufacturer last, and the manufacturer estimates it had a solution within half an hour of being asked.
At 14:27 Appendix D reads: "Auto processing of flight plans recommences – technical system restored." The elapsed column says 05:55.
How this was checked. Every quotation was checked word for word against the saved text of the report named beside it, and every figure is quoted from those reports. The only arithmetic is the elapsed time between two printed times, and the timeline's elapsed figures are the panel's own. No individual is named: the reports name roles, and so does this piece.
Chain of Consciousness keeps a tamper-evident, dated record in hash-linked entries, so when a fallback started, who was called and when, and what fixed it can be read back after an incident instead of reconstructed from memory.
pip install chain-of-consciousness npm install chain-of-consciousness
Or start without installing anything: Hosted Chain of Consciousness.