OpenAI says 30% of SWE-Bench Pro is broken and drops its recommendation

A leading coding benchmark loses OpenAI’s backing

OpenAI has put a major warning label on SWE-Bench Pro, one of the public benchmarks used to compare advanced coding models and coding agents. In a post on X, the company said it audited SWE-Bench Pro and found that it no longer reliably measures frontier coding capability. The central number is sharp: OpenAI says 30% of the public tasks are broken, and it is retracting its previous recommendation that researchers use the benchmark as a leading coding evaluation.

"30% of SWE-Bench Pro tasks to be broken"

The failure mode described in the thread is not just normal leaderboard noise. OpenAI says some correct solutions fail because of hidden requirements, contradictory instructions, overly strict tests or incomplete grading criteria. For coding agents, that matters because benchmark scores often mix model capability with the behavior of the harness, tests and patch validation. If a task has requirements that are not visible to the model or a grader rejects valid fixes, a better agent can look worse than a weaker one.

OpenAI’s account is normally a product, model and research channel, so a benchmark audit from that account carries weight in the same ecosystem that uses these scores for model launches. The tweet links to an OpenAI analysis page, but that page was blocked by a JavaScript and cookie gate in this environment, so this article relies on the visible tweet text, public timestamp, engagement, and the stated 30% finding.

The practical effect is wider than one leaderboard. Coding-agent vendors increasingly market small differences across SWE-Bench-style tasks, while enterprise buyers use those comparisons as shorthand for developer productivity. A public retraction from OpenAI pushes evaluators to ask whether a score reflects real bug fixing, benchmark familiarity, or quirks in hidden tests.

Watch for three next steps: whether SWE-Bench Pro maintainers flag or remove the broken tasks, whether other coding benchmarks receive similar audits, and whether OpenAI publishes a replacement protocol that separates useful signal from grading noise.

Source tweet