Cognition's AI Productivity Guarantee pays when an index computed by its own estimator falls short of a customer's spend, and its guarantee page says Cognition can change the method with notice. The figure that validates the estimator, a correlation of 0.74, cannot see the level the money turns on.
On 4 June 2026 Cognition offered its enterprise customers a guarantee on Devin, its autonomous AI software engineer: "if Devin delivers less engineering value than you’re paying for, Cognition will fund your usage up to $10M until it does." A second post the same day describes the estimator that measures that value. Its headline figure is a correlation, r_log, of 0.74 on 233 held-out sessions.
The guarantee has the shape of a parametric contract: it pays on an index, here estimated engineering hours converted to dollars, and not on a loss anyone measured at the customer. That raises two questions. Who computes the index? And can the number that validates the estimator see the part of the index the money turns on?
On the first, every input to the index that we can name is Cognition's: the estimator, its calibration, the classifier that decides which sessions count as unproductive, and a dollar rate that the posts, as saved, do not publish. Cognition's guarantee page adds one more: asked whether Cognition can update the measurement methodology, it answers, "Yes, with reasonable advance notice." On the second, the answer is no. A correlation cannot see level, the overall size of the estimates, and the guarantee pays on level.
The guarantee post, by Scott Wu, gives the mechanism in two sentences: "Engineering hours are converted to dollar value using a standard global rate and compared against each customer’s actual consumption near the end of their annual contract. If the value falls short, we issue credits up to $10M."
The guarantee's hours are the estimator's hours. The post introduces the estimator, says Cognition validated it, and continues: "The results made us confident enough to offer a guarantee to our enterprise customers," followed by the terms. The guarantee page the post links to, devin.ai/guarantee, says it outright: "A value estimator runs on every completed Devin task, estimating the equivalent human engineering effort in hours." Its FAQ says the estimator was validated by "126 engineers across 8 enterprise customers"; the estimator post counts 126 users.
The post's "How it works" section says an agent reviews each completed Devin session and estimates how long a human engineer would have taken. "If the session resulted in unmerged PRs or was classified as otherwise unproductive, the output is considered not useful." The estimator post, by The Cognition Team, supplies the human reference: "We collected the data via live interviews and a survey." That gave 258 sessions from 126 users, of which 25 went to a development set and 233 were held out.
As published on 4 June 2026 and saved by us on 2 November 2026, the two posts do not state the value of the rate, whether the $10M is a cap per customer or for the whole program, how a credit is sized once the value "falls short," or any audit, dispute process or third-party check. The guarantee page, which we read on 2 November 2026, settles one of these: "If the value falls short, we issue free usage credits to cover the gap, up to $10M." It does not give the rate, say whether the cap is per customer or in total, or describe an audit, dispute process or third-party check. Cognition's Master Services Agreement, last updated 8 June 2026 on our copy, does not mention the guarantee either. It ranks the Order Form first in a conflict (section 12.12), so we infer that any guarantee terms sit in Order Forms, which we could not read.
In the three World Bank catastrophe bonds we read, a named firm is the calculation agent. The Chile 2023 case study describes $630 million of earthquake cover. Payouts follow "the physical parameters of an earthquake event," so "there is no need to wait for incurred losses to be assessed before the insurance would payout." After a qualifying event, "Chile will issue a notice to the independent calculation agent (AIR Worldwide) who will determine the payout amount based on the location, magnitude, and depth of an earthquake." The World Bank pays "as soon as a calculation report is available" (p.1).
The Mexico 2024 and Jamaica 2024 studies name the same firm in the same job: "AIR Worldwide as risk modeler and calculation agent." In all three, the World Bank "procured external service providers" and AIR Worldwide is one of them. The firm modeled the risk too, but it is neither the government that receives the money nor the World Bank that issued the bond.
The three case studies are summaries, not prospectuses. Only the Chile study uses the word "independent," and none of the three says whether the government or the investors can dispute a calculation.
Amazon's Compute SLA for EC2 is the closer precedent, because there the vendor runs the process. A customer opens a case in the AWS Support Center and, for a region-level claim, supplies "your request logs that document the errors and corroborate your claimed outage." Then: "If a claim under the Region-Level or Instance-Level SLA is confirmed by us as valid, we will issue you a Service Credit within one billing cycle following the month in which your request is confirmed by us." The remedy is credits and nothing else: "Service Credits will not entitle you to any refund or other payment from AWS." In the SLA and in the guarantee, the vendor decides and pays in its own credits.
Two things separate them. AWS publishes its schedule: credits of 10%, 30% and 100% of that month's EC2 bill in the affected region as monthly uptime falls below 99.99%, 99.0% and 95.0%. Cognition's credit is the gap itself, sized by its own index and capped at $10M. And AWS's trigger is an outage, which the SLA expects the customer to document from its own side. The guarantee's trigger is a model's estimate of the hours a human would have needed, and the posts, as saved, do not say whether a customer can see the per-session estimates behind its total. The guarantee page says "A real-time dashboard is available so you can track progress at any time," without saying whether it shows the estimate for each session.
The page also says who can change the index. Asked "Can Cognition update the measurement methodology?", it answers: "Yes, with reasonable advance notice." What has been shown stays shown: "Historical data on the dashboard is immutable," and once calculated and displayed, "it does not change retroactively." The page names no role for the customer or a third party in a change of method.
Basis risk, the subject of Malawi Paid $5 Million for Drought Insurance, applies to this index too, and Cognition's threats-to-validity section concedes it: "We measure engineering capacity, not whether that capacity was deployed on high-value work." This piece leaves that gap aside.
The estimator post validates the model with r_log, which it does not define in the text we saved. We read it as the correlation between the logarithms of the model's hours and the engineers' hours.
A correlation does not change when a variable is rescaled. Take the model's raw estimate m, in hours, and a calibration of the form h = a × m^b, which is the form Cognition uses. Then log h = log a + b × log m, which shifts and stretches log m, and a correlation ignores both. Every positive a and b gives the same r_log, so a calibration that halves every estimate and one that multiplies every estimate by ten both leave the 0.74 at 0.74.
Our script draws 233 synthetic sessions, the size of Cognition's held-out set, and applies five calibrations. The numbers are ours, not Devin's. They show a property of correlation and make no claim about Devin.
| calibration applied to m | r_log | summed estimate ÷ summed synthetic "human" hours |
|---|---|---|
| none | 0.854596 | 0.334 |
| 2.08 × m | 0.854596 | 0.694 |
| 2.28 × m^0.923 | 0.854596 | 0.618 |
| 0.5 × m | 0.854596 | 0.167 |
| 10 × m | 0.854596 | 3.337 |
The correlation is identical to six decimals. The sums, which are what a guarantee pays on, run from 0.167 to 3.337 times the synthetic engineers' total, and the last two rows differ by a factor of twenty.
Cognition's estimator post holds a version of this. It reports that "Our initial, uncalibrated model consistently underestimated," fits a correction in log space, h = 2.28 × m^0.923, and then says that constraining the slope to one, a single multiplicative constant of 2.08, "changes every metric by at most 0.01." On the validation sample, the two calibrations are hard to tell apart. On a single session they are not.
At a model estimate of 1 hour the power fit gives 2.28 hours and the constant gives 2.08. The two cross at 3.29 hours. At 100 hours the power fit gives 159.93 hours against the constant's 208.00. Against the constant, the power fit is 9.6% higher at 1 hour and 23.1% lower at 100 hours. A customer's total mixes these ratios by session length, so the choice of calibration moves a customer with long sessions one way and a customer with short sessions the other. That is our arithmetic on the post's two formulas.
The 0.74 sees none of this, and that is no flaw in the statistic. It answers whether the model's estimates move with the engineers'. The payout asks how big the sum is.
The blindness reaches the reference too. The calibration is fitted to the engineers' own hours, and the post's threats section says "Our ground truth is self-reported. In our video user interviews, users were aware they were talking to Cognition, which might bias them." Scale all of those hours by a constant and r_log stays at 0.74, while the fitted a, and with it the level of every estimate, scales by the same constant. That is our arithmetic on a log-space fit.
The estimator post is open about the level. Its introduction says the model has an r_log of 0.74 and "appears to be unbiased," and, as the evaluation section shows, both are claims about logarithms. After calibration, the total of the human estimates is still 1.4× the total of the corrected model estimates, and the post explains: "This gap is expected: an estimator that is unbiased in log-space becomes biased once its predictions are summed in linear space, systematically underestimating the total." It chooses not to close the gap: "we report the unadjusted figure as a deliberately conservative underestimate."
The explanation fits the size. If log errors are normal with spread s, the expected true total exceeds the sum of the estimates by a factor of exp(s²/2). Setting that to 1.4 gives s = 0.820, a typical error of 2.27× and 60% of sessions within a factor of two. The post reports around 50% and calls 2 to 3× errors common. That is our arithmetic, under log-normal errors. The post, as saved, gives no interval for the 1.4×.
The posts, as saved, do not say whether the guarantee's index carries the 1.4×. They say the estimator reports the unadjusted figure, and they do not say whether the guarantee uses that figure or a corrected one. Suppose it uses the unadjusted figure. Then the index, in aggregate, is the engineers' total divided by 1.4, and the guarantee pays whenever the engineers' total is below 1.4 times the customer's spend. On $1,000,000 of spend:
| engineers' value, as a multiple of spend | index | shortfall |
|---|---|---|
| 0.8 | $571,429 | $428,571 |
| 1.0 | $714,286 | $285,714 |
| 1.2 | $857,143 | $142,857 |
| 1.4 | $1,000,000 | $0 |
The shortfall column takes the credit to equal the shortfall, as the guarantee page describes it: "free usage credits to cover the gap, up to $10M." On this reading, a customer whose engineers value the work at 20% above spend still shows a shortfall of $142,857, which fits the estimator post's description of a system "calibrated to underestimate rather than overestimate delivered output." The conservatism favors the buyer.
It also depends on the rate. The cushion is a ratio of hours and the payout is in dollars. A rate 1.4 times higher raises the index by the same factor and cancels the cushion exactly: the 1.0 row above, a shortfall of $285,714, becomes $0. So does a calibration level 1.4 times higher.
Two more choices lower the index: sessions with pull requests but none merged are discarded, which the estimator post says it did because "we wanted to err conservative," and the classifier for sessions with no pull request removes around 1 to 20 percent of sessions, "depending on the customer." The posts, as saved, publish the calibration, the pooled 1.4× and that range. They do not publish the rate, a customer's own exclusion share, or which version of each the guarantee uses. We have no evidence that Cognition sets any of them to offset another. The posts, as saved, also give a buyer no way to check that it does not.
From the posts as saved, a buyer cannot audit this index. Four requests would change that.
Nothing we could read shows whether any of the four is already in an Order Form. The guarantee post's last sentence is "Existing customers can reach out to their account team." The questions go there.
Show your work. estimator_arithmetic_c10765.py does the arithmetic above with Python's standard library alone, from the figures in Cognition's two posts:
Page numbers are the PDFs' own.
Not found or not read.
Part of Where the Number Came From, on how a published number is a fact about the way it was measured: Stanford says 12% to 66%, but 12% of what? · The safety score is a fact about the test rig · Tracing the 2026 AI-failure statistics to a primary
Chain of Consciousness keeps a dated, tamper-evident record in hash-linked entries. Log each estimate beside the method version that produced it, and a change of method shows on the record, not only in the total.
pip install chain-of-consciousness npm install chain-of-consciousness
Or start without installing anything: Hosted Chain of Consciousness.