Separating Signal from Noise: Flaws in Coding Evaluations Exposed

OpenAI has conducted an extensive audit of SWE-Bench Pro, a widely used coding benchmark, and found widespread task issues. The investigation reveals that approximately 30% of the tasks are broken, which can lead to inaccurate measurements of model capabilities and misrepresent safety cases. This is particularly concerning for OpenAI’s Preparedness Framework, where accurate evaluations are crucial for deployment and safety decisions.

The audit follows a similar review of SWE-bench Verified, another popular coding benchmark, which was found to have fundamental design and contamination issues. At the time, OpenAI recommended switching to SWE-Bench Pro as an improvement over its predecessor. However, this latest investigation has uncovered significant problems with SWE-Bench Pro.

SWE-Bench Pro was designed to test models on longer horizons and more realistic coding tasks, aiming to better track agentic coding capabilities. Tasks are sourced programmatically from public and private repositories, requiring models to implement a solution that passes new tests for a feature without breaking existing functionality. The 731-task public split saw frontier models improve from a pass rate of 23.3% to 80.3% in eight months.

OpenAI’s audit involved reviewing the dataset using a datapoint analysis pipeline, which flagged likely evaluation flaws based on model attempts at tasks, task metadata, and failure traces. Each flagged task was then assessed through multiple investigator-agent passes and independently reviewed by five experienced software engineers, with disagreements escalated for further investigation.

The issues primarily fell into four categories: overly strict tests that enforce specific implementation details not specified in the prompt; underspecified prompts that omit requirements hidden in tests or are not reasonably inferable; low-coverage tests under check the requested feature, allowing incomplete fixes to pass; and misleading prompts that point models toward incorrect behavior or contradict test requirements.

The audit’s findings highlight the difficulty of curating hard but fair benchmarks. OpenAI estimates that approximately 30% of SWE-bench Pro tasks are broken, advising model developers to carefully examine results. The aim is to ensure that task failures reflect genuine model limitations and successes represent complete and valid solutions to prompt requirements.

To assess data quality in evaluations, OpenAI created a quality assurance pipeline. This pipeline reviews the instructions given to models, their attempts at solving tasks, and tests used for grading these attempts to flag likely broken or problematic examples. An initial automated filter flagged 286 potentially broken tasks, which were then reviewed through human-supervised agent review and human annotation campaigns.

The human reviewers were more likely than investigator agents to mark tasks as broken, with some disagreement on categories between the two review paths. However, in no flagged task was ‘not broken’ the most common human label. The overlap between reviewer judgments and agent pipeline flags was 74%, indicating that both methods captured similar failure modes.

Compared to the agent pipeline, human reviewers were more likely to select multiple labels for a task, suggesting additional or overlapping issues not identified by agents. This resulted in conservative labeling: capturing broad failure modes while undercounting cases where reviewers saw extra problems. The largest difference was in low-coverage tests, which humans selected as the most common issue for 9.4% of the benchmark compared to 4.1% from the agent pipeline.

The issues identified highlight the importance of rigorously checking benchmarks. Problem descriptions and merged code often do not align due to long back-and-forths between maintainers and contributors in open-source repositories. Tests included in pull requests can be overly strict, written to validate specific changes rather than defining implementation-agnostic standards for solving tasks.

Evaluation flaws are now easier to detect with improved model capabilities allowing deeper inspection of prompts, tests, patches, traces, and edge cases. This helps surface benchmark issues previously costly or impractical to find at scale. OpenAI hopes the wider evaluation community will develop new benchmarks built by experienced software developers specifically to test model capabilities.

This approach can preserve high standards for realism while ensuring better human oversight throughout the process. Given the issues uncovered in this analysis, OpenAI retracts its earlier recommendation to adopt SWE-Bench Pro. Ultimately, evaluations should provide meaningful signal through hard-to-game, easy-to-trust benchmarks genuinely reflective of model capability or alignment.