OpenAI quietly updates its evaluation metrics for GPT-6 Astra, making changes that appear to favor Astra and continuing to revise other metrics after launch
AI Signal Decode
OpenAI's post-launch adjustments to GPT-6 Astra's evaluation metrics, particularly its hallucination rate and mathematical performance scores, have fueled accusations of "benchmaxxing." The company states these changes are necessary refinements to ensure "best estimate of available model performance," attributing discrepancies to the inherent "noise" in evaluation setups, including model checkpoints, configurations, and harnesses. However, the sequential revisions, some benefiting Astra and others temporarily disadvantaging competitors like Anthropic's Fable 5.1, suggest a dynamic process of optimizing reported figures, raising concerns about the transparency and reliability of benchmark data in the highly competitive AI landscape.
The market implications of these metric alterations are significant, impacting how investors, developers, and users perceive the relative strengths of leading AI models. By adjusting benchmark results, OpenAI can influence market perception and maintain a competitive edge, especially when rivals' scores are also revised. The debate over benchmark accuracy, previously seen with Meta's Llama 4, underscores a broader industry challenge: standard evaluations are prone to gaming, making it difficult to ascertain genuine technological advancements versus optimized reporting. This necessitates a critical approach to AI performance claims, looking beyond headline numbers to understand the methodologies and conditions under which results were achieved.
Technically, the ability to significantly alter evaluation scores post-publication points to the sensitivity of benchmark tests to minute changes in parameters, such as harness configurations and reasoning levels. OpenAI's own admission that different harnesses can lead to vastly different results (e.g., 99.9% vs. 63% on ARC-AGI-3) illustrates this point. The practice of "benchmaxxing" exploits this sensitivity. Future developments should focus on more robust, immutable, and transparent evaluation frameworks. It is crucial to watch how OpenAI addresses these concerns, whether through more detailed system cards, independent verification, or standardized, less volatile benchmarks, to rebuild trust in AI performance reporting.