SWE-bench
The standard benchmark for AI coding agents on real GitHub issues — 500 verified tasks.
About
SWE-bench is the field's canonical benchmark for AI software-engineering agents: each task hands the agent a real GitHub issue from an open-source Python repository, and it passes only if its generated patch turns the repo's test suite green. The human-validated SWE-bench Verified subset (500 tasks) became the leaderboard every coding-agent vendor quotes — from GitHub's agent mode to frontier model launches — making it the de facto yardstick for practical coding ability.
Honest caveats, which matter more than the scores: a 2026 audit found 138 flawed tasks (over 60% unsolvable as written due to environment breakage and unrelated test failures), OpenAI stopped reporting SWE-bench scores in early 2026 and recommended alternatives like SWE-bench Pro, and the public issues and patches sit in every frontier model's training data, raising contamination concerns. Vals AI archived its standardized board in September 2026 after seven models hit 95%+, declaring the benchmark saturated. Treat it as a historical reference and sanity check, not a live differentiator.
What it can do
- 2,294 real-world GitHub issue tasks across Python repos
- SWE-bench Verified: 500 human-validated tasks
- Standardized patch-generation evaluation with test-based scoring
- Public leaderboard tracking 100+ models and agents
- SWE-bench Pro and Multimodal variants for harder, fresher tasks
Setup
Clone the SWE-bench harness from GitHub and run evaluations locally or via the official leaderboard.
Prerequisites
Python environment with Docker for the official evaluation harness.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 70; Trending score 85/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.