It is 3:14 a.m. somewhere. An on-call engineer’s phone buzzes for the fourth time that night. They thumb the screen, see p95 latency above 800 ms for service-foo for 90 seconds, mutter something, and roll back over. By Friday they will have received 47 pages, of which 22 were false positives, six were repeats of the same gradual cascade, and three required actual intervention. Their dashboard told them everything and nothing. They will keep showing up because they are paid to. The system will keep paging them because it does not know how to do anything else.
Most people in this position have already noticed something is wrong with the way we monitor things, and they have struggled to articulate it. The vocabulary for what is wrong does exist. It was published in 1977, in a book about how cities sound, by a Canadian composer who was not thinking about software at all. The vocabulary works anyway, because the underlying problem — distinguishing the meaningful sound from the meaningless one in an environment full of both — is the same in a hospital ward and an emergency room as it is in a server farm. Once you have the words, you can see things on your dashboards you could not see before.
This essay is about borrowing R. Murray Schafer’s vocabulary and using it to look at production systems differently.
Vancouver, 1973
Schafer was a composer and music educator who held a position at Simon Fraser University in Burnaby, British Columbia, through most of the 1970s. He founded the World Soundscape Project there in 1971 with a small team of graduate students, intent on a study no academic discipline had quite owned yet: the sounds of human environments treated as cultural objects, the way painters treat landscapes. In 1973 the team produced a long-playing record called The Vancouver Soundscape, two sides of recordings of the city captured at specific places and times — Stanley Park at dawn, the harbor foghorn at midnight, the streetcar barn, the morning markets. Some of those recordings are now in archives because the sounds themselves do not exist anymore. The streetcar bell is gone. The foghorn was retired. The market on Granville Island moved.
Out of that work, in 1977, Schafer published The Tuning of the World, a book that proposed an entire vocabulary for the acoustic environment. The vocabulary’s most useful component is its three-part taxonomy of sound.
| Term | Definition | Example |
|---|---|---|
| Keynote | The continuous background sound of a place — the kind you stop hearing within a few minutes of arriving. Not noticed; the condition for noticing. | Refrigerator hum, distant traffic, breeze in a forest |
| Signal | A sound that demands attention — designed to break through whatever keynote you are operating in. | Alarm, siren, horn, train whistle |
| Soundmark | A sound that is locally distinctive, the way a landmark is visually distinctive. The sonic identity of a place. | A specific church bell, the foghorn at one particular bay |
Schafer also drew a distinction between hi-fi and lo-fi soundscapes. A hi-fi environment is one where individual sounds can be heard cleanly against the keynote — a rural meadow at dawn, where a single bird call carries across half a kilometer. A lo-fi environment is one where so much sound saturates the channel that nothing can be heard distinctly — a busy intersection, a factory floor. In a lo-fi environment, signals stop functioning as signals. Everyone is shouting. No one is heard.
Hold those four words in mind — keynote, signal, soundmark, lo-fi — and let me show you what your monitoring system has been trying to tell you.
Your Dashboard Is Mostly Signal
Every observability product I have ever used is designed almost entirely around signals. Alerts. Threshold breaches. Error rate spikes. Pager notifications. The product asks you to define a condition, set a level, and route a message when the condition crosses the level. PagerDuty was founded in 2009 explicitly to deliver signals more reliably. Datadog’s marketing pages are organized around the kinds of signals you can capture. Sentry is a signal pipeline.
This is not a mistake — signals are necessary. The mistake is that signals are all most systems capture, while the keynote — the continuous baseline against which a signal is detectable at all — is left almost entirely uninstrumented. Most teams track latency percentiles, but only as inputs to threshold alerts. Almost nobody plots the shape of the latency distribution over time and looks for changes in the shape itself. The 2025 work on what people are starting to call “gradual drift detection” — the openobserve.ai writeup is a good summary, as is the recent arXiv paper on the Boiling Frog Threshold (arXiv:2603.08455) — frames the problem precisely: anomaly detectors whose baselines auto-adjust to recent data will miss drifts in which each small change falls within the recently-normal range. The system’s own monitoring adapts to the slowly-worsening state, and the alert never fires.
The Boiling Frog paper’s example I keep coming back to: a vision system whose camera lens fogs over the course of a week. No single image is outside the “normal” range against the previous day’s images. By Friday the system cannot see, and the dashboard has been green for five days. This is the keynote shifting under instrumentation that only listens for signals.
A keynote shift is also, in Schafer’s hearing of cities, the most consequential change you can have. A city whose traffic hum drops by twenty percent has had something fundamental happen to it — a strike, a holiday, an evacuation. You will not hear it as such; you will hear it as the city sounds different today. Your nervous system will note that something is off well before you can articulate what. Production systems have the same property and almost nothing that captures it. The hum of healthy traffic is the most informative signal you are not collecting.
The Lo-fi Disaster
The second thing Schafer’s vocabulary names is the disease most observability programs already have. In a hospital ward, the soundscape is lo-fi: cardiac monitors, IV pumps, ventilators, infusion alarms, bed alarms, nurse-call buttons. The ECRI Institute, an independent patient-safety organization, has ranked alarm fatigue as one of the top three health technology hazards every year since 2012; in several of those years it has been ranked first. There are documented cases in the patient-safety literature of preventable deaths in which the cardiac alarm was sounding correctly and was ignored because everything was sounding all the time. In 2010 a Massachusetts General Hospital patient died after his cardiac alarm was turned to zero volume to stop it from bothering the staff. The hospital and the Boston Globe both reported on the case, and ECRI has cited it as an exemplar ever since.
The numbers from the software side are not much better.
| Source | Finding |
|---|---|
| Microsoft / Omdia, State of the SOC 2026 | Security alert false-positive rate: 46%. Average 2,992 alerts per organization per day. 63% go unaddressed. |
| Dr. Droid (observability vendor) | Sober internal target: >30% actionable alerts as a healthy ceiling. |
| Google, Site Reliability Engineering | Maximum recommended: 2 actionable pages per on-call shift. Empirical baseline at most companies: hundreds of pages per week. |
The industry’s aspirational signal-to-noise ratio is one that would be considered a hostile listening environment by anyone in acoustic ecology. The gap between the recommended state and the observed state is the gap between hi-fi and lo-fi.
A lo-fi soundscape cannot transmit signals. This is the part most people miss. Adding another alert to a system that already drowns its operators is not defense in depth; it is one more drop of saturation in a channel that has already lost its capacity to carry meaning. The operator’s nervous system stops responding to each individual alarm at a measurable rate. The patient dies with the alarm sounding. The outage extends an extra ninety minutes because the page that came in at 3:14 looked like the false ones at 1:02 and 2:31.
The way out of a lo-fi soundscape is not louder signals. It is fewer signals, paired with better keynote awareness. The operator who has trained their ear to hear the keynote shift before any signal fires is the one who responds in time, and they are not common, because we have not built either the tools or the language to teach the listening.
Your Stack’s Soundmarks
The third concept, and the one I find most underexploited, is the soundmark. Every production system has a small set of distinctive failure dialects — error patterns that only happen on that stack, that database, that load balancer, that particular legacy service nobody quite understands. Senior engineers know these by feel. They have heard them often enough to recognize them within a few seconds. That kind of timeout means the connection pool is bleeding again. That cascading 502 means the new ingress is failing open to the old one. That memory growth pattern means the cache eviction is wrong but it’ll self-correct in 20 minutes if we don’t touch it. These dialects are the soundmarks of the system — locally distinctive, community-defining, captured almost entirely in the heads of a few people.
Standard monitoring tools do not encode soundmarks. Their rules are generic. They alert on p99 latency over a threshold, not on the specific timing pattern of your specific stack’s specific failure mode that the engineer who left the company two years ago could read in five seconds. That knowledge is not in your dashboards. It is in a runbook nobody reads, or it is gone.
Schafer’s worry about soundmark loss in cities — the church bell, the foghorn, the streetcar bell — is the same as the engineering manager’s worry about institutional knowledge walking out the door. The system’s distinctive sounds are part of its identity, and when they are not preserved, the system becomes generic in a specific bad way: the team cannot tell this outage apart from any other outage of the same surface shape. They lose the ability to respond fast because they no longer recognize what they are hearing.
The practical version of soundmark preservation in observability is something like a structured catalog of named failure patterns, each with its symptoms, its candidate causes, its first-response actions, and its history. Some places call this a runbook. The good ones call it a pattern library. The best ones treat it as living documentation that engineers contribute to whenever they notice a new dialect. The act of naming the pattern — giving it a memorable, team-internal handle — is half the work, because once it has a name, you can train people to hear it.
Listening Is Trained
The final concept, which Schafer drew on but Jonathan Sterne developed more rigorously in his 2003 book The Audible Past, is the idea that listening itself is not a natural capacity. It is a trained one. Sterne traces how the practice of medical auscultation — the use of a stethoscope to diagnose by sound — was an entire pedagogical revolution in the early nineteenth century, requiring a new kind of attention that did not exist before the instrument. Doctors had to learn to listen in a structured way. The interpretation of heart sounds and lung sounds is a discipline with its own vocabulary, its own examination standards, its own decades of pedagogy. A first-year medical student does not hear what a senior cardiologist hears, even pointing the same stethoscope at the same chest.
The observability parallel is precise and underappreciated. The 2026 New Relic AI Impact Report (cited by Rootly) found that AI-enabled incident-response teams averaged 26.75 minutes per incident, while non-AI teams required 50.23 minutes — a near-2× speedup. That gap is partly tooling, but a large share of it is what the senior operator notices that the junior operator does not. The senior has trained their ear on this stack’s keynote. They hear the shift before any threshold breaches. They know the soundmarks of the system the way a regular knows the regulars at a bar.
This is a teachable skill. It is not currently taught.
What to Do on Monday
If you are responsible for a production system, three habits worth installing.
One: audit your monitoring for keynote capture. Most systems track signals well and keynotes badly. Pick the three or four metrics that constitute the hum of a healthy version of your service. Latency distribution shape, not just percentile. Traffic composition by client type. Queue depth distribution. Cache hit ratio. Plot the shape of these over time. Do not alert on them; observe them. The next time you have a sense that something is off without being able to point to a fired alarm, look at the keynote plots. The shift will usually be visible there.
Two: measure your soundscape’s fidelity. Count your false-positive rate, your unaddressed-alert rate, and your pages-per-shift. If your aspirations are below the Microsoft/Omdia baseline of 46% false positive — and they have every reason to be — set a real target and remove alerts until you reach it. The hardest culture work in observability is convincing a team that deleting alerts is professional progress, not negligence. The lo-fi soundscape is the negligence.
Three: name your soundmarks. Build a living document — a wiki page, a Notion library, a directory of markdown files — where each known distinctive failure pattern has a name, a description, its candidate causes, a first-response action, and a date of last occurrence. Update it whenever a new dialect is recognized. Pull it up during onboarding. Use it during incident response. The naming is the training.
The deepest claim of acoustic ecology, and the one Schafer pushed hardest, is that what you can name, you can attend to; what you cannot name, you cannot hear. The vocabulary of keynote, signal, soundmark, and lo-fi is not new to your work. You have been operating inside it for years. The vocabulary is what most observability vendors lack, and the gift Schafer left, almost half a century ago, is that the vocabulary is already there. You only have to borrow it.
Sources: R. Murray Schafer, The Tuning of the World (1977); The Vancouver Soundscape LP (World Soundscape Project, 1973); Jonathan Sterne, The Audible Past (Duke University Press, 2003); ECRI Institute, annual Top Health Technology Hazards lists (alarm fatigue ranked top three every year since 2012); Massachusetts General Hospital cardiac-alarm fatality (2010), Boston Globe coverage; Microsoft & Omdia, State of the SOC 2026; Google Site Reliability Engineering (O’Reilly, 2016), chapters on alert design and on-call load; Dr. Droid public observability commentary on actionable-alert targets; openobserve.ai gradual-drift-detection writeup (2025); “Boiling Frog Threshold” arXiv:2603.08455; New Relic AI Impact Report (2026) via Rootly.
Soundmark Preservation Is Signed-History Preservation
Acoustic ecology says: preserve soundmarks the way historical societies preserve buildings. Observability needs the same discipline for failure patterns — an append-only, signed history of every named dialect, its first-response action, its occurrences, and the engineer who recognized it. When the engineer who could read that timeout pattern in five seconds leaves, the recognition is supposed to leave with them. With a chain, it doesn’t. The soundmark stays in the catalog. The next operator can hear it.
pip install chain-of-consciousness
npm install chain-of-consciousness
Try Hosted CoC — structural memory and audit trail for systems where the senior operator’s ear is the last copy of the knowledge.