By METR surveyed 349 technical workers about AI productivity gains and found a median self-reported 2x increase in the value of their work. Then METR quietly warned readers to be skeptical of the numbers. Their own staff — the same researchers who have spent years running controlled trials on AI’s actual impact — reported the lowest productivity gains of any group in the survey. That contradiction is worth sitting with.
What the Survey Found
The study, published May 11, 2026, covered 87 software engineers alongside researchers, academics, and founders. The headline number is compelling: developers report 1.4x to 2x more value from their work due to AI tools. Self-reported speed gains are even higher — a median 3x — though METR explicitly notes that speed overstates value and asked the value question intentionally.
The trajectory is striking. Respondents retrospectively peg their AI value gain at 1.3x in March 2025, 2x now in March 2026, and forecast 2.5x by March 2027. If you take these numbers at face value, AI tools are compounding year over year. Most survey respondents apparently do take them at face value.
METR’s own staff do not.
The Ghost of the 2025 Study
In mid-2025, METR ran a rigorous randomized controlled trial with 16 experienced open-source developers on large, mature codebases — the kind of real-world condition that matters. With full access to Cursor Pro and Claude 3.5/3.7 Sonnet, the developers took 19% longer on tasks than without AI assistance. Not faster. Slower.
The perception gap was staggering. Before starting, developers predicted AI would cut their time by 24%. After completing tasks, they believed AI had sped them up by 20%. The actual result was the opposite. This is not a small rounding error — it is a documented gap between how AI tools feel and what they do. ByteIota covered this finding in depth when it first broke last year.
That study is why METR researchers distrust their own survey results. When you have seen controlled evidence of AI slowing developers down, a self-reported 2x value increase is hard to accept without scrutiny.
Why the Survey Might Still Be Right
Both data points can be partially accurate. The 2025 controlled trial had a significant methodological issue that METR acknowledged in February 2026: developers who most heavily depend on AI were opting out. They did not want to do complex tasks without their tools, so they declined to participate. This means the study systematically excluded the tasks where AI provides the most value — exactly the use cases the survey picks up.
There is also a timing difference. The 2025 study used Claude 3.5 and 3.7 Sonnet. By early 2026, Anthropic had released Claude Opus 4.7 with an 87.6% SWE-bench score, Cursor 3 shipped parallel agent workers, and the entire ecosystem matured substantially. A study run today would measure different tools.
Jellyfish’s 2026 State of Engineering Management report adds an objective data point: teams in the top quartile of AI adoption have 2x the PR throughput of low adopters. That is not self-reported perception — that is commit data. High-adoption teams are shipping more. Whether that represents more value is a different question.
The Measurement Problem Nobody Is Solving
The deeper story here is that nobody has good metrics. Harness surveyed 700 engineers and found that 81% of engineering leaders report code review time increased after deploying AI tools. Hidden overhead — developers sifting through AI-generated code they half-trust — consumes roughly a third of the work day in ways that never show up in velocity dashboards. Yet 89% of the same leaders believe their metrics accurately capture AI’s impact.
METR’s survey is the most honest attempt yet at measuring AI’s value effect rather than just task speed. But the researchers themselves flag reasons to discount their own findings. That combination — useful data, built-in skepticism — is rare and worth respecting. The AI code quality crisis compounds the picture: more code, faster, does not mean better outcomes.
What to Do With This
Stop trusting self-reported productivity numbers, including your own. The 2025 controlled trial showed developers believe they are faster when they are not. The 2026 survey shows they report large value gains that METR’s most informed researchers doubt. The right move is not to dismiss AI tools — Jellyfish’s throughput data and the raw trajectory both suggest real gains are happening. But until your organization has objective before-and-after metrics, any ROI claim on AI tooling is a guess dressed up as data.
METR’s May 2026 survey is valuable precisely because it comes with its own asterisk. Not many vendors attach asterisks to their productivity claims.
