OpenAI says 30% of SWE-Bench Pro is broken and drops its recommendation
Original: OpenAI finds 30% of SWE-Bench Pro broken and retracts benchmark recommendation View original →
A leading coding benchmark loses OpenAI’s backing
OpenAI has put a major warning label on SWE-Bench Pro, one of the public benchmarks used to compare advanced coding models and coding agents. In a post on X, the company said it audited SWE-Bench Pro and found that it no longer reliably measures frontier coding capability. The central number is sharp: OpenAI says 30% of the public tasks are broken, and it is retracting its previous recommendation that researchers use the benchmark as a leading coding evaluation.
"30% of SWE-Bench Pro tasks to be broken"
The failure mode described in the thread is not just normal leaderboard noise. OpenAI says some correct solutions fail because of hidden requirements, contradictory instructions, overly strict tests or incomplete grading criteria. For coding agents, that matters because benchmark scores often mix model capability with the behavior of the harness, tests and patch validation. If a task has requirements that are not visible to the model or a grader rejects valid fixes, a better agent can look worse than a weaker one.
OpenAI’s account is normally a product, model and research channel, so a benchmark audit from that account carries weight in the same ecosystem that uses these scores for model launches. The tweet links to an OpenAI analysis page, but that page was blocked by a JavaScript and cookie gate in this environment, so this article relies on the visible tweet text, public timestamp, engagement, and the stated 30% finding.
The practical effect is wider than one leaderboard. Coding-agent vendors increasingly market small differences across SWE-Bench-style tasks, while enterprise buyers use those comparisons as shorthand for developer productivity. A public retraction from OpenAI pushes evaluators to ask whether a score reflects real bug fixing, benchmark familiarity, or quirks in hidden tests.
Watch for three next steps: whether SWE-Bench Pro maintainers flag or remove the broken tasks, whether other coding benchmarks receive similar audits, and whether OpenAI publishes a replacement protocol that separates useful signal from grading noise.
Related Articles
HN treated OpenAI's post less as benchmark housekeeping and more as an obituary for a famous coding leaderboard. The thread cared far more about flawed tests and contamination than about who happened to top the chart first.
Why it matters: public coding benchmarks are getting less useful at the frontier, so a fresh product-side score can move developer attention fast. Cursor says GPT-5.5 is now its top model on CursorBench at 72.8% and is discounting usage by 50% through May 2.
LocalLLaMA’s reaction was almost resigned: of course the public benchmark got benchmaxxed. What mattered was seeing contamination and flawed tests laid out in numbers big enough that the old bragging rights no longer looked stable.