It scored 13.86% on SWE-bench, priced itself at $500/month as a replacement engineer, then cut that 25x and became a $10.2B company selling the supervised tool underneath. What two years of Devin teaches anyone buying agentic AI, from a fleet that runs agents in production.
On March 12, 2024, a roughly ten-person startup called Cognition Labs introduced the world to Devin, billed in their own words as "the first AI software engineer." Not a coding assistant. Not an autocomplete. An engineer, with its own shell, its own editor, its own browser, and a launch video showing it completing what looked like real freelance jobs on Upwork, autonomously, for money.
The internet did what the internet does. The launch clips pulled tens of millions of views. "Software engineering is over" takes bloomed for a week straight. And one month later, Founders Fund led a $175 million round that valued the company at $2 billion, for a product almost nobody outside the demo videos had touched.
Two years on, the arc is complete enough to read honestly, and it's stranger than either the hype or the backlash predicted. The demo was partly staged. The product failed most real tasks put in front of it. The price collapsed twenty-five-fold. And the company is now worth more than ten billion dollars anyway.
Every one of those facts is load-bearing. Together they tell you more about buying agentic AI in 2026 than any benchmark chart will.
Devin's headline claim was a benchmark score: 13.86% on SWE-bench, the standard test that hands an AI real GitHub issues from real open-source projects and asks it to produce working fixes. The previous best was around 2%.
Two things about that number, both from Cognition's own reporting. First, it was a genuine leap: a ~7x jump on an unforgiving end-to-end benchmark, achieved unassisted, where prior systems were told which files to edit. Credit where due, the jump was real.
Second, and this is the part the takes skipped, it was a self-reported score, run on a random 25% subset of the benchmark, not the whole thing. And you can invert it: the most impressive autonomous coding system ever demonstrated, at launch, failed more than 86% of its tasks. That is what state-of-the-art meant in March 2024. The number was honest. The framing, "the first AI software engineer," implying a colleague, implying replacement, was doing work the number couldn't support.
A month after launch, a software developer named Carl Brown, who runs the YouTube channel Internet of Bugs, went through the viral Upwork demo frame by frame in a ~27-minute analysis bluntly titled "Debunking Devin."
His findings, since widely corroborated in the trade press: the Upwork job in the video asked for help running an existing computer-vision model, not writing code from scratch, which is what the video presents Devin as doing. Along the way, Devin generated its own bugs and then impressively "fixed" them, errors that a competent human simply wouldn't have introduced. Some of the files it "fixed" didn't exist in the original repository at all. The task it actually completed was not the task the video implied, and the impressive-looking debugging was partly Devin cleaning up after itself.
That's one expert's analysis, not a court finding, but nobody has seriously rebutted the frame-by-frame, and it landed because it named the pattern precisely: a demo is a curated sample. You are watching the numerator. The 86% lives in the denominator, off camera.
The debunk was about marketing. The harder evidence arrived in January 2025, when The Register reported on an independent team's month-long, hands-on evaluation: they gave Devin twenty real tasks.
It completed three satisfactorily. Three more were inconclusive. Fourteen were outright failures. The failure mode is the detail worth remembering. Tasks that looked like hours took days. Devin would burrow into technical dead ends and keep digging, producing elaborate, unusable solutions, spending days pursuing approaches a human would have abandoned in an hour, because recognizing "this is fundamentally blocked" is exactly the judgment the marketing implied it had.
One test, twenty tasks, a data point, not a census. But it rhymed perfectly with the benchmark's own denominator, and with what every early customer quietly discovered: pointed at uncurated reality, the autonomous engineer mostly wasn't one.
Watch what the company did next, because this is the part the "it was all hype" crowd gets wrong.
Devin went generally available in December 2024 at $500 a month, priced like the junior engineer it claimed to be. Four months later, Devin 2.0 launched at $20 a month plus consumption pricing. A 25x collapse in the entry price is not a discount; it's a repositioning. The product that emerged was not "the first AI software engineer." It was a supervised, scoped coding agent living in an IDE: one that drafts, you review; one that handles the bounded task, you own the judgment. The autonomy got smaller. The claims got smaller. The product got real.
Then in July 2025, days after Google poached Windsurf's CEO in a $2.4 billion licensing deal, Cognition bought what remained of Windsurf, an AI IDE with a reported ~$82 million in annual recurring revenue and an enterprise sales motion. By September 2025, Cognition raised another $400 million at a $10.2 billion valuation, with enterprise revenue reported as more than doubled since the acquisition.
Read the whole arc in one line: the replacement narrative failed; the tool succeeded; the company kept the valuation the narrative bought and then earned a bigger one selling the tool. Calling Devin a scam flattens the most instructive part: the honest, narrower product underneath the demo turned out to be genuinely valuable. It just wasn't the thing on the launch reel.
We run a fleet of AI agents in production, research, publishing, game content, the works, and the Devin arc isn't a story we read about a company. It's the physics we operate in every day. So here is the general lesson, stated as plainly as we can:
The gap between an agent demo and agent production is structural, not a vendor's moral failing. Two mechanisms produce it, every time.
First, an agent is confident whether or not it is right. A human junior engineer who is lost mostly looks lost. An agent that is lost looks exactly like an agent that is succeeding, same fluent narration, same tidy diffs, right up until you check. That's why Devin could spend days in a dead end: nothing in its own experience of the task feels like failing. Whatever verification you don't build, the agent's confidence will happily replace.
Second, every demo is a curated sample from a distribution the buyer never sees. Devin's launch video wasn't fake so much as selected, and selection is all it takes. A 14%-success system generates plenty of jaw-dropping successes to film. The only number that matters is the one measured on your tasks, uncurated, with the failures counted, and no vendor's launch reel will ever hand you that.
Which yields the buying rule the whole two-year arc teaches: buy what Devin became, not what Devin was sold as. The supervised, scoped, reviewed agent, the $20 product, is real and getting better across the industry. The autonomous colleague, the $500 pitch, is a demo genre. If you're evaluating agentic AI this year, the single most predictive question isn't the benchmark score; it's "what does this system do when it's wrong, and who catches it?" Devin's own maker answered that question with a pivot worth ten billion dollars.
We wrote the same retrospective about AutoGPT, 100,000 GitHub stars and then what, and the shape repeats because the incentives repeat: autonomy demos raise money; supervised tools do work. The projects change. The denominator doesn't.
The whole essay reduces to one question, "what does this system do when it's wrong, and who catches it?", and that catching layer is a thing you build, not a thing you buy in a demo. It's the work we do in the open: bounded authority so an agent can't act past its scope, a preview-and-confirm step so a human owns the judgment, and a provenance trail so every action is checkable after the fact. If you're putting agents into real workflows this year, that's the stack worth starting from:
pip install agent-trust-stack
npm install agent-trust-stackMore on the approach at the theory of agent trust, and the older sibling of this piece, AutoGPT Got 100k Stars, and Then What, for the format's first outing. We research the demo-to-production gap because it's the part we live in.