SWE-Bench Pro V2: A Cleaner, Harder-to-Game Leaderboard

Open source benchmarks are easy to criticize and hard to refresh, but alongside human authored benchmarks like SWE Atlas, they give you a fuller and more reliable picture of how software engineering agents perform, across both real-world conditions and controlled tests.

1. Introducing SWE-Bench Pro V2

Today we are releasing SWE-Bench Pro V2, a refreshed public split of SWE-Bench Pro with a modified benchmark and a network-locked evaluation protocol, together with a refreshed public leaderboard. We partnered with Reflection to co-develop this benchmark.

The release will be available on HF here and the task directories, verifier, and evaluation harness configuration here.

Improvements include:

• SWE-Bench Pro v2 consists of 642 tasks across 11 repositories, down from the original 731 public instances. We dropped 89 tasks which were found to be invalid during our review process.
• SWE-Bench Pro v2 Hard consists of 51 tasks from the above 642 that we found to be most challenging for various models.
• We refreshed all tasks and environments, including the task instructions, verifiers, and images. We ran several rounds of human expert and agentic review to fix issues such as reward hacking, task underspecification and ambiguity, solution leakage, and other general quality issues.
• 211 tasks were improved to have better dependency support for running OSS harnesses.

How we approached the refresh

A leaderboard is only as reliable as its tasks, quality controls, and evaluation infrastructure. Drawing on community feedback and our own validation, we reviewed every task, rebuilt the release pipeline around explicit quality gates, audited trajectories from earlier runs, and addressed the sources of leakage and grading risk we identified. We hold our leaderboards to a high standard and value the community feedback that helps strengthen them. We will continue to publish our findings and improvements transparently.

2. Current standings after the refresh

Public vs. private

Explaining the gap. We noted a significant gap between public and public eval results - e.g., 17.8% for Opus 5. The public runs were network-locked and audited, with no successful retrieval from code hosts or module proxies, making evaluation-time leakage an unlikely explanation. More plausible contribution is Training-time exposure. The public repositories, their fixing commits, and the benchmark itself have been on the open web since before these models were trained. The locked protocol cannot remove what a model already memorised; near-ceiling public scores are consistent with both genuine capability and recall. The private set is the only clean measurement of that, and it is why we maintain one.

Across the public leaderboard, frontier models - including Claude Opus 5 (99.4%), Kimi K3 (97.7%), and GPT-Astra (96.9%) demonstrate strong performance across the full 642-task suite. To complement our core benchmark, we introduced the SWE-bench Pro v2 Hard: a dedicated collection of 51 high-difficulty tasks grounded in verified failure modes from top frontier runs. By centering on deep multi-file dependencies, subtle system bugs, and long-horizon architectural reasoning, this split provides a sharp, high-rigor testbed designed to measure the next frontier of SWE agent capabilities.

3. What we changed

Evaluation hardening: anti-cheating measures

An agent benchmark is only as good as the ways it can be gamed. V2 closes four channels, and each closure was verified from inside the sandbox, not assumed from the config.

• Git history and hidden files. Every image is built from a sanitised bundle, so the fixing commit, stray refs, stashes, hooks, and test files are simply not there. A probe that checks this from inside the agent sandbox ships with the release and runs before every evaluation.
• Network. During the agent phase the sandbox can reach the model endpoint and nothing else. No code hosts, no package indexes, no web-fetch tools. Setup and grading phases keep their normal access. We verified the lock under the exact flags used for the runs; connections are refused at connect time.
• Grading. The grader no longer runs in the sandbox the agent just used. Every agent's diff is captured and re-graded on a pristine image, and both grades are published. This matters in practice: it caught one frontier model forging a Go module version and its checksum, and another editing dependencies directly in the module cache. Both passed in place and failed on a clean image.
• Verifier. Across all 642 tasks, hidden-test paths are restored or removed before the test patch is applied, agent edits to fixtures and snapshots are reverted (with a per-task allow-list for data the reference patch legitimately changes), stale Python bytecode is purged, and a leftover git lock can no longer make restores fail silently. Ten tasks also got environment fixes the verifier needed, such as kernel headers or a Go module proxy.

The human expert effort behind V2

The corrections in V2 were made by 23 contracted engineers, each working in a language they write professionally, across all 11 repositories in the split. They spent 1,897 hours across the public and private sets, a median of about two hours per task touched. Every instruction they rewrote was then handed to a blind reader: a second engineer from the same pool who had never seen the task. The blind reader received only the corrected instruction, implemented against it in the task's own container, and was graded by the task's hidden tests. Every instruction correction in V2 was validated by someone who solved the task from the text alone.

Benchmark repair is fundamentally an expert data-quality problem. Automated checks can surface suspicious tasks, but determining whether a failure comes from an ambiguous specification, a faulty test, a broken environment, or an incorrect implementation requires engineers who can understand and debug the repository end to end.

This work followed a multi-stage quality process. One expert diagnosed and corrected each task; a second expert, without access to the original instruction or reference solution, then implemented the corrected specification in the task’s own container. If that implementation did not pass the hidden tests, the task returned for further review. Every instruction correction included in V2 was ultimately validated by an engineer who solved the task from the text alone.

4. How do modern LLM harnesses affect model behavior?

This study is contributed by Reflection.

We ran open-weight models (GLM-5.3, Kimi K3 and Inkling) through mini-swe-agent, Pi and OpenCode, on all tasks in SWE-Bench Pro v2 under the same locked budget and no network beyond the model endpoint.

All trajectories in this study will be made publicly available.

*Note: All harnesses were run at the same settings except mini where we kept the 30s bash timeout imposed by it. No meaningful turn or token limits were imposed.

The open-weights models showed strong generalizability and got similar performance across the collection (except Inkling which performs lower on Pi). The harnesses differ in interesting ways: mini-swe-agent gives the model a single bash tool and kills any command that runs longer than 30 seconds. Pi and OpenCode give structured read, search and edit tools alongside a shell.

So we focus this study on how the model behavior changed: token spend, time spent per task and the footprint of failures.

How different harnesses spend tokens per task

While the change in accuracy is very small for Kimi-K3 and GLM-5.3, the swing in cost is bigger.

Kimi-K3 under Pi is the leanest combination in the study, spending least tokens per solved task.

When we look at failure traces, we observe these don’t happen because the model works less. At the median, failed runs use more steps and tokens than solved runs (in eight of nine cases) and about twice the reasoning. Here’s a breakdown of how they spend time in various rollouts.

How different harnesses spend time per task

Where the time goes depends on the model more than on the tools. GLM-5.3 spends the largest share of its wall-clock running tests under mini-swe-agent and Pi; Inkling spends the largest share of its reading and searching files.

For example, the 30-second command cap in mini-swe-agent shows how the models adapt: a Go build or a jest suite does not finish in time, so GLM-5.3 and Kimi K3 learn to launch it in the background and poll, with commands like sleep 29; cat /tmp/build.log; pgrep -f "go build"

Inkling lets it ride to cut-off and moves on to the next step.

What the harness changes, and what it does not

A harness is a system prompt plus a set of tools, and the models adapt to it: different paths, different reasoning lengths, different use of the tools.

What it does not change, for the strongest models, is the outcome. GLM-5.3 and Kimi K3 have on-par performance across the options we tested; when the same model solves a task under one harness and fails it under another, the two runs are indistinguishable. Pick the harness for the bill, not the score.

5. Key takeaways

• We’re releasing SWE-Bench Pro v2, a refreshed version of SWE-Bench Pro. Same repositories, same tasks minus the ones that were invalid, same hidden tests. What changed is that the numbers can now be trusted: the benchmark passes a two-sided gate (every task's reference patch passes and an empty patch fails) and the protocol closes the channels that were open improving the data quality and the evaluation process.
• Collaboration with the industry: During our development, we learned that on many tasks harbor fails to install a harness and attributes to a false negative. This is something we would love to improve across multiple evals in the industry.
• We listen. The wider community pushed us here; the probe scripts, capture and re-grade agents, per-task grades in both modes, and trajectories are released so that anyone can check our work. More to come.

Acknowledgements

Thank you to the community and everyone who contributed to the release: Lucy Ogaz, Yunzhong He, Ivan Guevara, Fernando Carabedo, Ignacio Ramet, Ezequiel Manoukian, Donnahue Sterling, Weijun Luo, Jose Bernal, Julio Reyna, Christopher Wang, Luis Leandro, Yannis He, Bing Liu, Emre Guven