Scale AI released SWE-Bench Pro V2. It is a refreshed public split with a locked evaluation protocol. The update drops 89 invalid tasks and corrects 69 with contradictory instructions. The agent phase now reaches only the model endpoint with web tools disabled. Every agent diff is re-graded on a pristine image. A two-sided gate runs before release to catch regressions. 211 tasks got better dependency support and 10 got environment fixes. Two residuals remain open. The benchmark evaluates AI agents on contamination-resistant software engineering tasks. It sources tasks from diverse and complex codebases. Top models score around 23% on the public set. Performance drops further on the private subset. Models show varying success rates by programming language. Repository-specific difficulty impacts model performance.
What changed
SWE-Bench Pro V1 had 731 tasks, 89 of which were invalid and 69 had contradictory instructions.
What it unlocks
Evaluating AI agents on contamination-resistant, industrially relevant software engineering tasks with locked protocols.
- 642 tasks across 11 repositories
- 23.3% top resolve rate on public set
- 22.7% to 17.8% drop on private subset
- 107.4 mean lines of code changed per task
Sources