SWE-bench

Dataset Summary SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution. The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Want to run inference now? This dataset only contains the problem statement… See the full description on the dataset page:

Source: Hugging Face Hub (princeton-nlp/SWE-bench). Metadata imported from the dataset’s Hub tags.