AI Agent Glossary · AI Coding Tools Glossary
SWE-bench
A benchmark of 2,294 real GitHub issues from 12 Python repositories; a model passes if its patch makes the repository's tests pass.
Released by Princeton in late 2023, SWE-bench replaced toy function-writing tasks with real issue resolution: read the issue, navigate the codebase, write a patch, and the hidden test suite decides. SWE-agent scored 12.5% at launch in March 2024; by 2025 the human-validated Verified subset was effectively saturated. It became the reference number in every model launch, and is now succeeded by SWE-bench Pro.