OpenAI has published a closely watched benchmark note arguing that model performance on difficult agent tests can hinge as much on harness design as on the model itself. In a post about ARC-AGI-3, the company said that enabling two settings in its Responses API—retained reasoning and compaction—lifted GPT-5.6 Sol’s score on the public task set from 13.3% to 38.3%, while also cutting output tokens by about sixfold.

The result matters beyond one puzzle benchmark. ARC-AGI-3 is designed to test whether AI agents can learn unfamiliar 2D games from interaction rather than explicit instruction. OpenAI’s claim is that the benchmark’s default setup was suppressing performance by discarding internal reasoning between actions and by trimming older context as sessions grew. For AI builders and buyers, that turns a benchmark story into a deployment story: the way a model is wrapped, prompted, and allowed to manage memory can materially change what it can do.

What OpenAI changed in the ARC-AGI-3 setup

According to OpenAI, the company initially saw surprisingly weak results from GPT-5.6 Sol on ARC-AGI-3. The model scored 7.8% in one earlier framing discussed by the company, and OpenAI said GPT-5.5 was even lower. On the public set using the official harness, OpenAI now reports GPT-5.6 Sol at 13.3%.

The company says it investigated why a model that had shown strong performance in other game-like tasks looked ineffective here. Its conclusion was that the default ARC-AGI-3 harness did not preserve the model’s private reasoning across turns and used a rolling truncation scheme that gradually removed older interactions from view.

OpenAI’s alternative implementation used the Responses API in a way that mirrors how it says models are deployed in ChatGPT and Codex. The first setting, retained reasoning, keeps prior private reasoning available across tool calls and turns when the previous response ID is passed forward. The second, compaction, summarizes and compresses history rather than simply dropping older context when limits are reached.

With those two changes enabled, OpenAI says GPT-5.6 Sol reached 38.3% on the ARC-AGI-3 public set. The company also says output token use fell by 6x, which it attributes to the model spending less time re-deriving the game state on every move.

Why ARC-AGI-3 exposed a broader benchmark problem

ARC-AGI-3 is intended to measure adaptive reasoning in unfamiliar environments. Models receive text representations of game frames and level information, then must infer rules through trial and error. OpenAI notes that the benchmark’s maintainers chose a generic harness on purpose, aiming to make model weaknesses easier to see and comparisons fairer across systems.

That design goal creates a tension now common in AI evaluation. A generic harness can standardize testing, but it can also diverge from how leading model providers actually train and serve their systems. OpenAI’s argument is that ARC-AGI-3 was effectively testing not just the model, but a memory-constrained interaction pattern that differs from production use in ChatGPT and Codex.

This is not a trivial distinction. Many current AI agents depend on persistent state, long-horizon planning, and context management to perform reliably. If a benchmark removes those capabilities for all participants, it may improve comparability. But if one model family is explicitly trained to use retained reasoning and summarized context, the benchmark may understate how that system behaves in real tools.

At the same time, OpenAI’s post also underlines the opposite risk: benchmark improvements achieved through harness choices can blur the line between measuring the model and measuring the surrounding software stack. For enterprises comparing models, that means leaderboard numbers may say less than expected unless evaluation conditions match the intended deployment.

Evidence, caveats, and what is vendor-reported

The strongest claims in this story are vendor-reported. OpenAI is the primary source for the 13.3% to 38.3% score increase, the 6x output-token reduction, and the explanation that retained reasoning and compaction caused the gain.

OpenAI also says ARC-AGI-3 scores are measured using Relative Human Action Efficiency, or RHAE, and cites official gameplay logs to estimate that the average human tester scored 48%. That human estimate is presented by OpenAI as an estimate rather than a fresh benchmark result.

The company further argues that with its harness, GPT-5.6 Sol solves all six levels in a game shown on the leaderboard, while “no frontier model” solves any level beyond the first in the default leaderboard framing. Because the article is OpenAI’s own analysis of the benchmark and harness, readers should treat those comparative statements as claims from the vendor unless independently reproduced.

There are other limitations. The article does not provide an outside replication, a neutral re-scoring across multiple labs, or a head-to-head comparison showing whether similar memory-preserving changes would also lift rival models by comparable amounts. It also does not settle the normative question of what ARC-AGI-3 should measure: raw model behavior in a constrained generic interface, or performance in a provider-optimized agent stack.

Still, even with those caveats, the underlying point is credible and increasingly important. API settings, context retention, truncation policy, and tool orchestration can each change agent behavior in measurable ways. OpenAI is unusual here mainly because it quantified the effect in a public benchmark note.

What this means for AI builders and enterprise teams

For product teams building AI agents, the practical lesson is that memory architecture is no longer a back-office implementation detail. If a system repeatedly loses plans, observations, or prior hypotheses, performance can collapse on multi-step tasks even when the base model is strong. OpenAI’s write-up suggests that retained reasoning is especially important when actions unfold over long sequences and the model must learn from earlier attempts.

For teams using the Responses API, OpenAI is effectively making a product argument: the company says these settings are not exotic tweaks but part of the normal operating pattern behind ChatGPT and Codex. If true, developers benchmarking models in stripped-down loops may be testing a configuration that the provider does not consider representative.

The token-efficiency claim may matter as much as the score gain. A 6x reduction in output tokens, if reproducible in other agent workloads, would affect both latency and cost. In enterprise AI deployments, long-running workflows often fail not just because of reasoning quality but because they become slow, expensive, or brittle as context grows. Compaction, in OpenAI’s telling, improves both continuity and efficiency by avoiding the blunt lossiness of rolling truncation.

The story also has implications for evaluation practice. Buyers comparing systems for enterprise AI use should ask vendors not only which model was tested, but which harness, memory settings, tool protocol, and context policy were used. Benchmarks that do not specify those layers clearly may be less portable to real procurement decisions than they appear.

Competition and the benchmark politics behind the post

OpenAI’s post arrives amid rising pressure on frontier model labs to explain not just how models score, but why they score the way they do. Benchmarks increasingly shape narratives around model leadership, yet many agent benchmarks depend on wrappers and operational choices outside the core model.

By highlighting retained reasoning and compaction, OpenAI is also staking out a broader position: that modern AI performance should be evaluated as a system property, not just a weights-only property. That framing benefits vendors with tightly integrated products such as ChatGPT, Codex, and proprietary serving stacks, because they can argue that the customer buys end-to-end behavior, not an abstract base model.

That position will not be universally accepted. Some researchers prefer austere benchmarks precisely because they expose weaknesses that product tuning can mask. Others will argue that if users deploy models with memory, summaries, and tool loops, evaluations should reflect that reality. The ARC-AGI-3 debate is likely to be one of several flashpoints where those philosophies collide.

What to watch next

The most important follow-up signal is independent replication. If third parties reproduce OpenAI’s ARC-AGI-3 gains using the same Responses API settings, the finding will look less like a one-off vendor blog and more like a benchmark methodology lesson.

A second signal is whether ARC-AGI-3 maintainers respond by adding alternative harness tracks, such as a generic baseline and a production-optimized agent track. That would preserve apples-to-apples comparisons while acknowledging that real-world systems rely on memory management.

Third, watch whether OpenAI extends the same argument to other evaluations. If retained reasoning and compaction materially improve additional long-horizon tasks, the implications for AI agents, coding assistant products, and workplace automation could be broader than this one benchmark.

Finally, enterprise teams should watch for clearer documentation from vendors on context handling. If model providers start publishing standard benchmark recipes for ChatGPT, Codex, GPT-5.6 Sol, and related systems, it may become easier to compare enterprise AI behavior in a way that maps to deployment.

Creati.ai perspective

OpenAI’s post is self-serving, but it spotlights a real issue in AI evaluation: benchmark scores are often treated as if they come from the model alone when they actually reflect a stack of design choices. For anyone building AI agents, the takeaway is straightforward. Memory retention, summarization policy, and tool-loop structure can be first-order product decisions, not implementation garnish.

The more strategic takeaway is for buyers of enterprise AI. Do not procure based on a headline model score without understanding the harness behind it. If OpenAI is right, then a poorly matched wrapper can make a capable model look weak, and a well-matched one can unlock large gains in both reliability and cost. That shifts competitive advantage toward vendors and teams that can align model behavior, serving infrastructure, and application design into a coherent system.