In December 2021, Mark Hannam and colleagues published a paper in Nature announcing something genuinely new: the first measurement of strong-field precession in a black-hole binary, read out of the gravitational-wave signal called GW200129. The binary's orbital plane was wobbling like a tilted top, they reported, at a rate ten orders of magnitude faster than anything previously measured from binary pulsars. A textbook prediction of general relativity, observed in the wild for the first time.
Six months later, in June 2022, Ethan Payne and colleagues published a paper with a title you rarely see in physics: "The curious case of GW200129." They had traced the evidence for precession to a stretch of data from the LIGO Livingston detector that contained a known instrumental glitch, one that had been modeled and subtracted before the analysis. Their finding, in their own words: the difference between a spin-precessing and a non-precessing interpretation of the signal "is smaller than the statistical and systematic uncertainty of the glitch subtraction." The support for the discovery depended on how a preprocessing step had been modeled. They were careful to add that their results do not disprove precession in GW200129. This was not a debunking. It was something more unsettling: a demonstration that the headline result and its absence both fit inside the error bar of the data cleaning.
Then in November 2023, a third group, Macas and colleagues, built a neural-network model of the Livingston detector's broadband noise, re-cleaned the data with it, re-ran the analysis, and reported that the evidence for precession survives.
Three careful teams. Three defensible modeling choices. Three years. The question is still open, and the honest summary is not that anyone blundered. It is that the discovery was smaller than the variance introduced by a modeling decision that no published confidence interval had priced.
If you build machine-learning systems or run analytics for a living, you should recognize the shape of this story, because your pipelines produce it constantly. You just don't have three rival teams re-analyzing your dashboard to make it visible.
Gravitational-wave astronomy has a beautiful, clarifying asymmetry at its center, and it is worth two paragraphs of actual physics.
When two black holes spiral together, the signal sweeps upward in frequency, the famous chirp. The rate of that sweep is controlled, at leading order, by a single combination of the two masses called the chirp mass. Because the sweep rate is stamped on the longest, loudest, best-measured stretch of the signal, the chirp mass comes out of the analysis pinned tight. It is the quantity the data itself grips.
But nobody's press release is about the chirp mass. The questions everyone wants answered are about the components: how heavy was each black hole, how fast was each spinning. And the split of the total into components rides on much subtler features of the waveform, features faint enough that the answer depends materially on which waveform model you used to ask. The field maintains several independent families of these models, with names like SEOBNR and IMRPhenom, each an elaborate approximation to equations that cannot be solved exactly, each calibrated against expensive numerical-relativity simulations.
So one analysis produces, side by side, a number the data pinned and numbers the model shaped, and both arrive wearing the same kind of error bar. The collaboration knows this, which is why its discipline is to run multiple waveform families on important events and compare. Sometimes the comparison is reassuring: the flagship paper on GW190521, the heaviest merger of its era, closes its abstract by reporting that three different waveform models and direct comparison to numerical relativity "yield consistent estimates" of the properties. That sentence matters, and any honest telling of this story includes it. The models often agree.
The trap is the word "often." A 2022 study that assessed waveform accuracy across the parameter space found that for loud events the models in some cases fail the field's own accuracy criterion, that the worse-performing models "are more likely to give inconsistent estimations," and that accuracy degrades in specific, nameable regimes: high spins, unequal masses, orbital planes edge-on to the line of sight. A 2024 paper on calibration uncertainty states the stakes plainly: if accuracy requirements are not met, "systematic errors could dominate over statistical uncertainties." In other words, whether your error bar is honest depends on where in parameter space you are standing, and the signal does not announce which regime you are in. You have to check, and checking means swapping models.
Two more findings from that literature deserve a technical reader's full attention. First, the field's own estimate is that waveform accuracy must improve by at least three orders of magnitude to keep up with next-generation detectors. The instrument is going to outrun the model by a factor of a thousand: as the data gets better, the statistical term in the error budget shrinks and the model-form term does not move, so the fraction of the reported uncertainty that is honest goes down as sensitivity goes up. Second, and worst: systematic differences between waveform models can masquerade as violations of general relativity. Underpriced model uncertainty does not present as a slightly-too-narrow interval. It presents as a discovery.
Now translate, because the mapping is not a metaphor; it is the same statistics wearing different units.
Every model you ship reports uncertainty from inside itself: a confidence interval from sampling noise, a posterior from one prior and one likelihood, a cross-validation spread from one architecture and one feature set. All of that is the statistical term. The model-form term, what changes when a different reasonable team makes different reasonable choices, appears nowhere in the report, because from inside a single model it is invisible by construction.
Statisticians have been saying this in plain language for a generation. The classic tutorial on Bayesian model averaging, published by Hoeting, Madigan, Raftery, and Volinsky in Statistical Science in 1999, says in its abstract that conditioning on a single selected model "ignores the uncertainty in model selection, leading to over-confident inferences and decisions that are more risky than one thinks they are." That sentence is twenty-seven years old. Most production dashboards still ship exactly the practice it warns against.
If you want the empirical version, there is a famous experiment. In 2018, Silberzahn, Uhlmann, and colleagues handed the same dataset and the same question to twenty-nine analysis teams, sixty-one analysts in all: are soccer referees more likely to give red cards to darker-skinned players? The teams' effect estimates ranged from 0.89 to 2.93 in odds-ratio units. Twenty of the twenty-nine found a statistically significant effect; nine found none. The twenty-nine analyses used twenty-one distinct combinations of covariates. And by the study's own analysis, neither the analysts' expertise nor their prior beliefs explained the spread. Every one of those teams could have published an interval, every interval would have looked precise, and the spread between teams was wider than any interval within one. The between-model variance was real, decisive for the conclusion, and invisible from inside every single analysis.
Machine learning has its own name for the underlying condition: the Rashomon effect, the routine situation in which many models fit the data about equally well while disagreeing about the internals, the framing Leo Breiman pushed into statistics in his "Two Cultures" essay in 2001. When the Rashomon set is large, the question "which features mattered" does not have a model-independent answer at all, and reporting one model's answer with one model's error bar is reporting a choice as if it were a measurement.
Which is exactly the chirp-mass asymmetry. Your pipeline's aggregate, the overall accuracy, the total lift, the topline conversion effect, is usually your chirp mass: pinned by the data, stable when you swap models, the thing you can actually defend. The decomposition, which segment drove it, which feature is responsible, which component of the funnel to fix, is usually your component masses: the thing everyone wants to act on, and the thing the model is quietly inventing. Both arrive in the same report, wearing the same decimal places.
The physicists' response to this situation is not despair, and it is not averaging the problem away. It is a set of habits any data organization could adopt this quarter.
Run more than one reasonable model, on purpose, on the questions that matter. Not a hyperparameter sweep inside one architecture, which is the within-model term again, but genuinely different reasonable choices: a different model family, a different covariate set, a different preprocessing of the same raw data. The GW200129 dispute is, at bottom, three teams doing exactly this to a data-cleaning step, which is where an uncomfortable amount of model-form variance lives in analytics too. Think about where your own glitch subtraction hides: the imputation rule for missing values, the threshold that defines an outlier and silently deletes it, the attribution window that decides which touch gets credit for a conversion, the deduplication logic that decides what counts as one user, the seasonality adjustment applied before anyone models anything. Each of those is a modeling choice made upstream of the model, each has defensible alternatives, and each propagates into your headline numbers while appearing in no error bar anywhere. The teams that re-ran GW200129 were not auditing the waveform models first; they were auditing the cleaning. Yours would deserve the same visit.
Then sort your outputs into two bins, and label them. Robust-across-models: report with confidence, act on it. Model-dependent: flag it, and either report the spread across models or stop reporting the quantity as if it were measured. The sorting is the deliverable. A chirp-mass-style report says: the total effect is real and stable under every reasonable analysis we ran; the segment attribution flips sign when we change the covariate set, so we are not shipping it as a finding.
Price the spread, don't hide it. When the models disagree, the spread between them is not noise to be embarrassed by; it is the honest size of your uncertainty, and it is information about where more data will and will not help. Remember the detector lesson: collecting more data shrinks the sampling term only. If your models disagree about the decomposition, a bigger sample will sharpen each model's wrong confidence without resolving the disagreement.
And keep the humility clause, because it is the part that keeps the rest honest: the ensemble prices the uncertainty; it does not dissolve it. GW200129 is still unresolved after three excellent papers. The multi-model discipline did not hand the field an answer. It handed the field an accurate picture of what is and is not known, which is what error bars were supposed to be for in the first place.
The practical takeaway fits in three questions to ask of any number your team is about to act on. Which of our reported quantities would survive a model swap, and which have we never tested that way? What is our chirp mass here, the thing the data actually pins? And what are our component masses, the decomposition we want so badly that we may be reading it off the model rather than the world? If nobody in the room knows which is which, the error bars on the slide are describing the sampling noise of a choice.
Sources: Hannam et al., "General-relativistic precession in a black-hole binary," Nature (arXiv:2112.11300, Dec 2021); Payne et al., "The curious case of GW200129" (arXiv:2206.11932, Jun 2022); Macas et al., "Revisiting the evidence for precession in GW200129 with machine learning noise mitigation" (arXiv:2311.09921, Nov 2023); LVK Collaboration, "Properties and astrophysical implications of the 150 Msun binary black hole merger GW190521" (arXiv:2009.01190, 2020); "Assessing the model waveform accuracy of gravitational waves" (arXiv:2205.08448, 2022); "Accounting for numerical-relativity calibration uncertainty" (arXiv:2410.16859, 2024); Hoeting, Madigan, Raftery & Volinsky, "Bayesian Model Averaging: A Tutorial," Statistical Science 14(4), 1999; Silberzahn et al., "Many Analysts, One Data Set," Advances in Methods and Practices in Psychological Science 1(3), 2018; Breiman, "Statistical Modeling: The Two Cultures," Statistical Science, 2001.
If a number in your stack drives a decision, the first question above is only answerable if somebody wrote down which choices produced it. The imputation rule, the outlier threshold, the attribution window: those are the glitch subtraction, and they are usually undocumented by the time anyone asks. Chain of Consciousness records that reasoning as a durable artifact, so a result can be re-derived and challenged rather than defended from memory.
pip install chain-of-consciousness
npm install chain-of-consciousness