OpenAI triples ARC-AGI-3 score by retaining agent reasoning

The striking change on ARC-AGI-3 came from the test harness, not a new model. In a July 29, 2026 report, OpenAI said GPT-5.6 Sol scored 13.3% on the ARC-AGI-3 public set with the official harness, then rose to 38.3% when the company used a Responses API harness that retained reasoning and enabled compaction. Output tokens fell by 6x at the same time.

ARC-AGI-3 measures how well agents can explore unfamiliar 2D puzzle games and infer their rules without explicit instructions. The score is Relative Human Action Efficiency, a metric comparing model behavior with a human baseline. OpenAI’s post says the estimated average human tester scored 48%, while no frontier model on the public leaderboard solves any level beyond the first. GPT-5.6 Sol initially looked weak as well, scoring only 7.8% in the first pass OpenAI described.

The main issue was memory. The official harness discarded private reasoning after each game action, so the model could see a record of past moves but not the plans or intermediate ideas that produced them. It also used rolling truncation: once the context exceeded 175,000 characters, older messages were removed. That meant the agent could lose both its earlier observations and its own strategic thread.

OpenAI rebuilt the run closer to how it deploys models in ChatGPT and Codex. Passing the previous response ID in the Responses API retains reasoning across turns, and compaction summarizes long context instead of simply dropping the oldest material. With those two settings, GPT-5.6 Sol spent less time reinterpreting each game and did a better job preserving learned rules across longer runs.

The broader stake is benchmark interpretation. A single leaderboard number can look like a clean model comparison, but agent performance often reflects the whole system: model weights, tool interface, memory policy, prompt design, and context management. On long-horizon tasks, a generic harness can understate what a deployed agent can do, while a tuned harness can make comparisons harder to interpret.

This does not settle ARC-AGI-3 performance across labs. It is OpenAI’s own implementation, and independent comparisons still matter. The useful question now is whether other frontier models gain similar ground when reasoning retention and compaction are standardized, or whether GPT-5.6 Sol benefits unusually from that setup.