OpenAI says SWE-Bench Pro no longer reliably measures frontier coding capability after finding 30% of its public tasks broken. The cited issues include hidden requirements, contradictory instructions, strict tests and incomplete grading criteria.
The episode matters because governments are trying to govern AI while using the same tools inside the drafting process. South Africa pulled its first national AI draft after fictitious references surfaced, scrapping a plan that would have created three new institutions and new incentive programs.
HN treated OpenAI's post less as benchmark housekeeping and more as an obituary for a famous coding leaderboard. The thread cared far more about flawed tests and contamination than about who happened to top the chart first.