Microsoft Research, Microsoft AI, and KAIST published ProgramDistill this week — and it doesn’t grade AI coding agents on whether they can fix a GitHub issue. It asks something harder: can an agent watch a working app, figure out what behavior is missing, and implement it without ever reading the source code? The best model tested, GPT-6 Astra, failed on more than half of the full-app reconstruction tasks. Claude Opus 5 failed on more than 70%. These are the same models scoring 49–89% on SWE-bench. The gap is not a rounding error.
What ProgramDistill Actually Tests
Most SWE benchmarks hand the agent a GitHub issue and ask it to write a fix. ProgramDistill doesn’t give the agent a spec. Instead, the agent gets two things: a fully working reference application (source code hidden) and an editable copy with functionality removed. The agent must explore the reference to understand the missing behavior, then implement it in the editable version. The test is replay-verified — the same interaction workflow must succeed on the agent’s implementation.
The benchmark auto-generated 4,063 tasks from 26 web applications without human annotation, covering everything from simple atomic repairs (single missing behavior) to full-application reconstruction from a minimal scaffold. Restoration depth ranges from 1 to 8, where depth 8 requires recovering a chain of interdependent behaviors — think of it as undoing a layered refactor without the git log.
The Results
Nine frontier coding agents were evaluated. The headline numbers for full-application reconstruction:
| Model | Cumulative Score | Cost per Task |
|---|---|---|
| GPT-6 Astra | 49.2% | $33.99 |
| Claude Opus 5 | 28.8% | $28.86 |
| GPT-5.6 Sol | 21.1% | $16.01 |
For partial-application reconstruction — where the agent starts with a masked repository rather than a blank scaffold — scores improve substantially. But the depth problem shows up fast. Astra goes from 100% at depth 1 down to 64% at depth 8. Opus 5 goes from 96% to 32%.
The Depth Collapse Is the Real Story
Performance degrading with complexity is expected. The specific shape of the collapse is not. As restoration depth increases from 1 to 8, the lines of code an agent must restore grow 9.3×. But here’s the part that matters: the number of reference observations agents made per target dropped from 34.60 to 8.46. The harder the task got, the less agents looked at the working reference. That is exactly backwards.
This isn’t a model intelligence problem — it’s an agent behavior problem. Agents that face complex, multi-step tasks tend to start editing earlier and observe less, when the correct response is the opposite. Task scaffolding, not raw model capability, is the lever here.
Why Most Failures Are Observation Failures
The paper breaks down why agents fail in full-application reconstruction:
- 59.2% of failures: the agent never observed the behavior it needed to implement
- 27.9%: observed but implemented with wrong state transitions
- 11.1%: observed but implemented incorrectly
- 1.8%: observed and implemented, but agent never triggered the implementation
The majority of failures are not coding errors. The agent simply didn’t explore enough of the working reference to know what was missing. SWE-bench-style benchmarks cannot catch this failure mode because they give the agent a written specification — exploration is never required. The full paper (arXiv:2609.18805) details the failure taxonomy across all nine agents.
Astra’s Strategy Explains Its Lead
GPT-6 Astra’s observation-intensive approach — averaging 96.3 observation steps per trajectory versus roughly 10 edits — is what separates it from the pack. Agents that look more, implement better. The lesson for developers prompting or configuring agents is direct: front-load exploration. Explicitly instruct the agent to characterize the reference behavior before touching any code.
The application-level variance is equally striking. Astra recovered 81.8% of MailHub’s workflows but only 5.9% of Baserow’s. Baserow is a no-code database tool with complex stateful interactions; MailHub is simpler. This means the right question isn’t “which model should I use?” but “does this model handle stateful, complex apps?” Right now, none of them do well.
What Developers Should Take From This
ProgramDistill’s benchmark gap — 88.6% on SWE-bench Pro Verified versus 49.2% on full-app reconstruction — is not a reason to abandon AI coding agents. It’s a reason to use them more precisely:
- Keep tasks shallow. Depth 1–3 tasks get 85–100% success across all tested models. Depth 7–8 tasks should be decomposed manually before you hand them off.
- Mandate observation phases. Add an explicit step in your agent workflow: “Explore and document all relevant behavior in the reference before writing any code.” Don’t rely on the model to decide when it has seen enough.
- Don’t trust SWE-bench rankings for agentic work. A model scoring 88% on issue-fixing benchmarks may solve fewer than half of your reference-guided automation tasks. Test on your actual use case.
The benchmark dataset and pipeline are available on the Microsoft debug-gym project page. If you’re building or evaluating coding agents for real-world use, these 4,063 tasks are more predictive of production performance than anything currently on the SWE-bench leaderboard. As AI Weekly noted: “The gap between an instruction-style SWE score and this one is where autonomous-coding pitches will now get pressure-tested.”

