BENCHMARK GUIDE
What is SWE bench Verified?
SWE bench tests whether a system can produce a patch for a real software issue. Verified is a selected set of 500 problems checked by engineers.
When it is useful
Use the resolved percentage to compare runs with matching tasks and evaluation conditions. Record the agent’s tools, model version, retry budget and test setup.
What it cannot tell you
A result measures a configured system on a particular set of issues. It does not mean the model can complete the same percentage of all software work.
Two runs using the same model can differ because one can search files, run tests or make more attempts. Label these settings before comparing their scores.
The chart shows selected published results with a source, snapshot date and tested configuration. Coverage differs by benchmark. Missing scores stay blank; demo values are no longer shown.
Sources
Developer reports describe their own results. This site has not independently reproduced them.
SWE bench documentation