Microsoft Research just published the first peer-reviewed field study to measure what actually happens when you give tens of thousands of engineers access to Claude Code and GitHub Copilot CLI — using real telemetry, not surveys. The headline: a sustained 24% increase in merged pull requests. The finding that should change how you plan your own rollout: skip-level peer behavior predicts adoption better than any organizational mandate ever will.
The Numbers
The study tracked Microsoft engineers across a 16-week post-rollout window beginning January 5, 2026. Adopters merged 24% more pull requests compared to baseline [95% CI: +14.5%, +33.7%], and the lift held across the entire observation period — no novelty fade. February showed +29.4%, March and April settled at +20%.
The dose-response data is what makes this study worth taking seriously. Engineers using the tools on three days per week saw a +15% PR lift. At five or more days per week, the number jumped to +50.1%. The relationship was monotonic — more usage, more benefit, no plateau in sight. That’s not what you’d expect from a tool that generates plausible-sounding noise.
Copilot CLI vs. Claude Code: The Comparison Engineers Actually Want
For single-tool users, Copilot CLI users saw a +24.9% PR lift versus +11.4% for Claude Code users — roughly 2.2x the throughput gain. That looks like a clear win for Copilot CLI, until you look at session depth.
Claude Code triggered 2.3x more file edits per session. The tool goes deeper into a codebase per interaction. Copilot CLI closes more PRs; Claude Code rewrites more of the file tree. Which one you want depends on what your bottleneck actually is. If you’re optimizing for shipping features at volume, Copilot CLI’s PR-per-day number is hard to argue with. If you’re refactoring a messy codebase or tackling multi-file changes, the per-session depth matters more.
Notably, for enterprise teams with compliance requirements, Copilot Enterprise still has an advantage: IP indemnity and centralized team management that Claude Code doesn’t currently offer.
Why Your Rollout Email Won’t Work
The adoption findings are where this study gets genuinely interesting. Career stage, tenure, and demographics were all weak predictors of who tried Copilot CLI. Social exposure was the dominant signal — by a wide margin.
Engineers whose skip-level peers had adoption rates above 25% showed 216% higher odds of trying the tool themselves. Direct manager use added +82% odds. Reviewer peer adoption added +54%. The pattern is consistent: people adopt AI coding tools because they see colleagues — especially senior ones above them in the org chart — using them visibly and getting results.
That has a direct implication for rollout strategy. Sending seat license announcements does nothing. Getting your staff engineers and senior ICs to use these tools publicly — and talk about what they built — is the actual adoption mechanism.
The Familiar Tools Trap
One counterintuitive finding deserves attention. Engineers who were already heavy IDE Copilot users were 49-83% more likely to try Copilot CLI after the rollout. So far so sensible — familiarity lowers the barrier to trying something new.
But those same engineers were 12-15% less likely to retain Copilot CLI after the first week. The study’s interpretation: familiar alternatives reduce sustained habit formation. If you already have an AI tool that works well enough, you don’t commit to the new one. Your existing Copilot-heavy teams may be your lowest-retention cohort for CLI agent adoption — worth knowing before you target them first.
What This Doesn’t Measure
The study authors are direct about the limits of using merged pull requests as the outcome metric. More PRs merged faster does not prove better software shipped. Technical debt, review quality, long-horizon architectural work, and code that takes longer than 28 days to merge are all outside this dataset.
The cost reality also deserves mention. One extreme case cited in the paper involves monthly token spend approaching $1.4 million for a single heavy user. Microsoft scaled back broad Claude Code access during the study period due to cost pressure. The 24% PR lift is real; whether it clears the bar for your token budget depends on how actively your engineers use the tools and what your per-PR value looks like.
What to Take From This
The Microsoft study is the most rigorous data available on enterprise AI coding agent ROI. Its conclusions are narrower than the press headlines suggest — throughput up, quality unknown, costs real — but they’re grounded in four months of actual engineer behavior. For engineering leaders evaluating rollout strategy: prioritize visible peer use over broad mandates, expect your active coders to retain tools and your casual users to churn, and pick your tool based on whether you need PR volume or per-session codebase depth. Read the full TechRepublic breakdown for additional context on what this means at organizational scale.
