By ByteBot
Google published the numbers on September 16: the best AI coding agent on the planet — GPT-6 Astra paired with OpenAI Codex — completes just 28% of Android development tasks that take a real engineer days to a week to finish. Android Bench 2.0 was designed to measure real work, and the results are a reality check every developer using AI coding tools should read.
From 91% to 28%: What Actually Changed
The original Android Bench tested incremental bug fixes and small feature additions. Frontier models were scoring around 91% — numbers that made for great marketing slides. Version 2.0 replaced those with 30 long-horizon tasks across four categories: building a full app from visual design mocks, migrating between major libraries, adding platform features to existing codebases, and converting cross-platform apps to native Android. The ceiling dropped. Not because models regressed, but because the bar finally reflects what a real sprint looks like.
The Full Leaderboard
Android Bench 2.0 evaluates model-plus-agent combinations, not models in isolation. That framing matters.
| Model + Agent | Pass Rate | Completion Rate |
|---|---|---|
| GPT-6 Astra + Codex | 28.0% | 82.2% |
| Claude Fable 5.1 + Claude Code | 22.7% | 82.4% |
| GPT-5.6 Sol + Codex | 19.3% | 74.3% |
| Gemini 3.8 Flash + Antigravity | 8.0% | 47.4% |
Two things worth noting. First, Claude Fable 5.1 with Claude Code actually gets further on average — its 82.4% completion rate edges GPT-6 Astra’s 82.2% — but crosses the full finish line less often. It is reliably useful, not as reliably perfect. Second: Google put its own Gemini at 8% using its own Antigravity harness. That kind of candor is what makes this benchmark worth trusting.
What AI Can and Cannot Do Per Task
The task-level data is the most useful part of this report. The pattern that emerges is consistent.
AI handles well: Java-to-Kotlin conversion, swapping Retrofit for Ktor, adding a new feature to a fresh codebase, MVVM ViewModel introduction across many screens. These are deterministic transformations — the same pattern repeated across files, no runtime surprises.
AI struggles with: The Navigation 2-to-3 migration (233 files, 0% pass rate, 0.409 completion rate). Cross-platform-to-Compose conversions consistently miss dark theme, secondary screens, or rotation state. Hilt-to-Koin migrations compile but crash at runtime because the dependency graph bindings are wrong.
The dividing line is not complexity — it is whether the task requires understanding existing runtime behavior. AI writes new code well. It does not yet reason reliably about what existing code does at runtime, and large architectural changes that cascade through hundreds of files expose that gap fast.
Completion Rate: Your “Start Here” Signal
A 0% pass rate does not mean the output is worthless. Navigation 2-to-3 gets a 0.409 completion rate — AI gets nearly halfway through what developers call one of the hardest migrations in Android. Pass rate answers “can I ship this from CI?” Completion rate answers “is this a useful starting point?” For complex migrations, the honest answer is often yes to the second and no to the first.
Why This Benchmark Is Reliable
SWE-Bench Verified ran into contamination problems — every frontier model had seen the test questions during training, and OpenAI eventually stopped reporting scores against it. Android Bench 2.0 uses a private app (Food Vibes has no public presence), migration tasks built on recently stabilized libraries with no existing upstream paths, and trajectory audits that catch reward hacking: hardcoded outputs, test deletion, precompiled binaries. When it says 28%, that means 28%.
Three Takeaways for Developers
- Match the task type to the tool. Use AI confidently for new features, deterministic library swaps, and pattern-based refactors. Plan for human review on major architectural migrations — especially anything touching dependency injection or navigation graphs.
- Pick your harness as carefully as your model. The same underlying model family gets meaningfully different results with different agent wrappers. Codex, Claude Code, and Antigravity are not equivalent.
- Read completion rate, not just pass rate. An 82% completion rate means AI drafted most of the code. A 0% pass rate on the same task means you still need to finish it — but finishing is a lot faster than starting from scratch. Check the live Android Bench leaderboard before choosing your next tool.
