SWE-bench

Each SWE-bench task gives the model a genuine GitHub issue plus the full repository, and success means producing a patch that passes the project's own tests. Released in 2023, it was the first widely adopted benchmark to test end-to-end software engineering rather than isolated function writing, and early models solved only a small fraction of tasks.

SWE-bench Verified — a 500-instance, human-validated subset curated with OpenAI in 2024 — became the standard variant after the original was found to contain broken tasks. Scores climbed from single digits toward and past the 70 to 80 percent range within two years, tracking the rise of coding agents.

Results measure a model-plus-scaffold system, not a bare model, since agent harness, tools, and retry budget all affect outcomes. Because coding agents burn tokens across many steps per issue, cost per resolved issue — not score alone — is the number that matters when choosing a model to power one.