By ByteBot
Meta researchers just dropped a benchmark that should make you rethink how you pick your AI coding agent. SWE-Together, built from 11,260 real developer-agent sessions, does not ask models to fix a GitHub issue and call it done — it tests how well they collaborate under pressure, with a user correcting mistakes in real time. The result flips the standard leaderboard ranking: Claude Opus 4.8 leads not on raw task completion alone, but by requiring the fewest corrections of any model tested, 1.38 per task, versus 2.17 for the worst performer. That gap is a proxy for how much of your day you spend babysitting your agent.
The Problem With SWE-bench
SWE-bench Verified is the metric most developers cite when comparing coding agents. Claude Opus 4.8 at 88.6%, GPT-5.5 at 88.7%, and so on. It is a reasonable benchmark for one thing: can the model fix a well-specified GitHub issue, alone, with no feedback, in one shot?
The problem is that nobody works like that. Real coding sessions involve clarifying what you actually meant, catching mistakes the model missed, redirecting after a wrong turn, and verifying that the fix does not break something else downstream. SWE-bench Verified scores none of that. Worse, OpenAI deprecated SWE-bench Verified from its own reporting in February 2026 after finding that frontier models could reproduce the benchmark’s gold patches verbatim from training data. GPT-5.2, Claude Opus 4.5, Gemini 3 Flash — all of them could recall answers to specific task IDs. The benchmark was not just measuring the wrong thing; it was partly measuring memorization.
What SWE-Together Does Instead
The Meta team started with 11,260 recorded real developer-agent coding sessions, then ran them through a three-stage quality filter. After checking for multiple user turns, actual code edits, clear repository context, and no dependency on live external services, only 109 tasks survived — 0.97% of the original corpus. These are not synthetic tasks. They are reconstructions of real sessions, with sandboxed environments and executable verification artifacts.
To replay each session against new models, the researchers built an LLM-based user simulator. Unlike earlier approaches with scripted feedback, this one is trajectory-conditioned: it watches what the agent actually does and responds based on that, rather than following a fixed schedule. In a blind Turing test, human annotators could not reliably distinguish the simulator responses from real user messages — it achieved a 46% human pass rate.
The benchmark tracks two metrics: pass@1 (did the code end up correct?) and User Correction (UC), counting how many corrective turns the simulated user had to make. Lower UC means the model got it right with less steering.
The Results
Claude Opus 4.8 led on both metrics: 63% pass@1 and 1.38 average user corrections per task. GPT-5.5 scored 58% pass@1 with a higher correction rate. MiniMax-2.7 sat at the bottom with the lowest pass@1 and the worst correction rate at 2.17 per task. Across all models, the researchers found a Pearson correlation of -0.92 between correction count and task success — stronger models genuinely require less hand-holding. This validates UC as a real proxy for capability, not noise.
The divergence between SWE-bench Verified rankings and SWE-Together rankings should give you pause. The headline scores you have been using to compare tools may not predict how much re-prompting you will actually do in a full workday.
The Correction Loop Is the Job
A companion study analyzed 20,574 real coding agent sessions across 1,639 repositories and found that 91.49% of agent failures require explicit user correction to resolve. Developers push back against agent outputs in 41% of turns. Only 44% of agent-written code makes it into final commits.
These numbers tell a consistent story: you are not using a coding agent to generate code and ship it. You are co-authoring with a model that will get things wrong, and your job is to steer it efficiently. An agent that completes 63% of tasks with 1.38 corrections is strictly more productive than one that completes 58% with 2.17 corrections — even though the raw task numbers look closer than they are in practice. Scale that to a full sprint, and the productivity gap becomes significant.
The Benchmark Landscape Is Shifting
SWE-Together is not alone. The same month saw SWE-INTERACT (arXiv 2606.30573), Dialogue-SWEBench, ClawMark, and SlopCodeBench, all pushing toward multi-turn, long-horizon evaluation. The research community has concluded that the single-shot paradigm was a useful proxy for early agents but does not hold up for today’s autonomous coding tools. SWE-bench Pro, with 1,865 tasks from private codebases that cannot be memorized, is already the new standard on official leaderboards. SWE-Together adds the dimension that Pro still lacks: the quality of the iterative loop.
What to Ask Before Picking Your Next Agent
The next time a vendor cites SWE-bench Verified, push for multi-turn results. Specifically: how many correction turns does the model require per task? Does it recover from its own mistakes, or does it need a hard reset? Can you see User Correction rates or equivalent metrics?
The tools that win in practice are not necessarily the ones with the highest single-shot benchmark scores. They are the ones that keep you in flow rather than pulling you out to correct, redirect, and restart. SWE-Together is the first benchmark that tries to measure that directly, and the gap between the best and worst models is already 57%. That is not a rounding error.
