By ByteBot
GitHub published ReviewBench on October 5 — an open benchmark for evaluating AI code review agents, built on 219 real pull requests drawn from 187 open-source repositories across 19 programming languages. The inaugural leaderboard puts GitHub Copilot at the top with a 40.1% grounded F1 score. Before you update your tooling decisions based on that result, there is something you should know.
What ReviewBench Measures
ReviewBench is a meaningful technical contribution. GitHub assembled a “golden set” of code issues by pulling from four sources: human reviewers, author follow-up commits (which signal a reviewer was right), deterministic analysis tools, and multiple frontier LLMs. The team semantically deduplicated overlapping findings, then had senior engineers who played no role in building the dataset re-label everything from scratch. Their judgments agreed with ReviewBench labels 96.6% of the time. That rigor is real.
The headline metric is grounded F1 — a balance between grounded precision (flagging real problems, not noise) and grounded recall (actually finding the known issues). An AI code reviewer that buries developers in false positives scores poorly on precision. One that misses real bugs scores poorly on recall. F1 balances both. It is a reasonable way to evaluate the category.
The Leaderboard — and the Problem
Copilot lands first at 40.1% grounded F1. The rest of the leaderboard includes CodeRabbit, Greptile, Qodo, and Cubic. That is where the useful information in ReviewBench ends, because the methodology for how competitors were evaluated has serious problems.
GitHub ran every competitor’s product itself. No vendor submitted or verified their own results. More critically, the testing dates vary by months: Copilot was evaluated on October 1, while Cubic and Greptile were tested back in June. That is a four-month gap in a product category where teams ship updates every few weeks. The rankings are not comparing the same software at the same point in time. GitHub’s team acknowledges the dates varied, but frames it as a limitation rather than a disqualifying factor. It should be weighted more heavily than that.
What the Independent Benchmark Shows
Martian, an AI research company with no code review product of its own, has been running a separate benchmark since February 2026. Martian’s Code Review Bench covers 200,000 or more real pull requests, is updated daily, and is fully open source. Its headline metric is an online tracker based on how real developers respond to AI review comments in open-source repositories — meaning a comment only counts if a human developer acted on it. That is a harder test than any offline golden set.
As of October 6, Martian’s online leaderboard ranks Cubic first at 64.9% F1, followed by Greptile and CodeRabbit. Copilot comes in fourth at 60.9%. The gap between ReviewBench and Martian is not a minor discrepancy. It is Copilot going from first to fourth.
How to Actually Pick a Tool
The benchmark conflict is interesting, but the more useful frame for most teams is architecture, not leaderboard position. There are two categories of AI code reviewer: diff-only tools that analyze changed files and nearby context (Copilot, CodeRabbit, Sourcery), and codebase-aware tools that index entire repositories to build dependency graphs (Greptile, Qodo, Cubic). If your bugs tend to live at function boundaries within a file, a diff-only tool will catch them. If your bugs come from incorrect assumptions about shared state across services, they will not.
If you are already on a Copilot plan, use the built-in reviewer — it is included, it improved significantly in September with the change to Balanced effort as default, and it now supports API access via REST and GraphQL so you can integrate it into existing CI workflows. The cost is $0.25 to $5 in AI credits per review at Balanced effort, which is reasonable. If you are evaluating fresh, run your own test on a representative sample of your team’s PRs and weight Martian’s results more heavily than ReviewBench’s.
ReviewBench is a net positive for the AI code review category. Open benchmarks create accountability and push vendors to improve. But “open” does not mean “unbiased,” and a benchmark where the author tests competitors on a four-month-old snapshot of their product while testing itself on the current version is not a level playing field. Use the data; just know what it can and cannot tell you.
