By ByteBot
Microsoft ran the largest peer-reviewed field study of AI coding agents ever published, and the headline number is a clean 24% lift in merged pull requests. But the study comes with a plot twist: Microsoft quietly cancelled most of its Claude Code licenses two months before the paper went live, because the token bill was unsustainable. Productivity gains are real. The business model is still broken.
The Study
The paper — arXiv:2607.01418, authored by Emerson Murphy-Hill, Jenna Butler, and Alexandra Savelieva at Microsoft Research — tracked tens of thousands of engineers across a 16-week window from January to April 2026. It examined two CLI-based agentic tools: Anthropic’s Claude Code and GitHub’s Copilot CLI. This is the first study to use developer-level telemetry to measure both adoption dynamics and output impact for this class of tool at enterprise scale.
The key metric is merged pull requests — an imperfect proxy, as the authors acknowledge, but the most reliable signal available at this scale. The 24% lift (95% CI: +14.5% to +33.7%, p<0.001) held steady across the full four-month observation window, a meaningful distinction from prior IDE tool research that consistently showed fade-out effects within weeks.
Copilot CLI Beat Claude Code — By a Lot
Here is where the numbers get uncomfortable for Anthropic. When the researchers isolated each tool’s contribution:
- Copilot CLI: +24.9% PR throughput [+23.0%, +26.8%]
- Claude Code: +11.4% PR throughput [+9.4%, +13.6%]
Copilot CLI delivered 2.2× the productivity lift, with a difference significant at p<0.0001. The likely explanation is structural: Copilot CLI is tightly integrated with the GitHub pull request workflow, so its gains show up directly in the PR throughput metric. Claude Code’s strengths — deep multi-file reasoning, complex refactors, autonomous long-running tasks — do not necessarily translate into more merged PRs within a standard sprint cycle. The tool wins on capability. It loses on this particular measurement.
There is also a dose-response relationship that matters. Engineers using agentic CLI tools three days per week saw +15% more merged PRs. At five or more days per week, that climbed to +50.1%. The more you use it, the more it pays off — but that same relationship means heavy users are generating dramatically more tokens, which is where the cost problem lives.
Who Benefits — and Who Does Not Adapt
The study found a C-shaped benefit curve by career stage. Junior engineers gained more than mid-level engineers, since there is more repetitive and boilerplate work to delegate. Senior engineers and managers also outperformed mid-level, because they used AI as a decomposition layer — breaking complex tasks into manageable chunks and running them in parallel. Mid-level engineers showed the lowest relative gains, likely because they carry the most ownership of complex, cross-team work that does not fit neatly into AI-assisted execution yet.
One retention finding is worth flagging for rollout planning: engineers who were already heavy IDE Copilot users had a higher chance of trying CLI tools but a 12–15% lower retention rate. The likely explanation is a mental model mismatch — they expected IDE-style autocomplete and encountered something fundamentally different. Onboarding for existing Copilot users needs to explicitly address this shift from inline completion to agentic delegation.
Adoption Spreads Through Peers, Not Procurement Emails
On adoption, the findings are unambiguous: formal rollout communications matter far less than peer visibility. When 25% or more of skip-level peers were already using the tool, an engineer’s odds of trying it were 216% higher. Direct manager usage added 82%. Reviewer peer adoption added 54%.
The deployment implication is specific: place highly visible engineers on early access, make their usage visible across the organisation, and let social proof do the work that onboarding decks cannot. Broad licence access without visible reference users is the slow path.
Then the Bill Arrived
The study’s publication date is awkward for Microsoft. By May 2026, the company had already decided to cancel Claude Code licences for its Experiences and Devices division — the team building Windows, Microsoft 365, Outlook, Teams, and Surface — effective 30 June. Around 5,000 engineers were directed to migrate to GitHub Copilot CLI instead.
The reason is straightforward: the pilot launched under flat seat licensing, which kept token consumption invisible. When Microsoft switched to usage-based billing, the true cost surfaced — between $500 and $2,000 per engineer per month. At that range, a 5,000-engineer cohort runs $2.5 million to $10 million monthly. That maths does not work at current pricing without governance.
Microsoft is not alone. Uber’s CTO said the company consumed its entire 2026 AI coding budget within four months. The FinOps Foundation’s 2026 State of FinOps report found that 73% of enterprises exceeded projected AI costs. The Microsoft study itself cites a single Meta employee who averaged 281 billion tokens in 30 days — a potential $1.4 million bill from one person.
What This Means
The productivity case for agentic CLI tools is now empirically established. A 24% throughput lift that persists over four months is not a vendor claim — it is peer-reviewed, at enterprise scale, with a credible methodology. The question is no longer whether these tools work.
The question is whether the pricing model makes them viable at enterprise scale without governance infrastructure. Rolling out Claude Code or Copilot CLI without per-user token limits, team-level cost attribution, and model routing matched to task complexity is how you end up with a productivity study and a cancelled licence in the same quarter. Microsoft just demonstrated exactly that sequence.
There is also a measurement gap worth naming directly. The study tracked throughput, not output quality. Merged PRs are not the same as correct, maintainable, production-ready code. The next study we need tracks defect rates and revert frequency alongside PR volume — because a 24% lift in throughput that comes with a hidden quality cost is a different headline entirely. For now, the case for adopting agentic coding tools is solid. The case for adopting them without a token budget is not.
