Signal 1 source · 38m ago

Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements

A recent analysis of AI pretraining progress from 2019 to 2025 reveals that advancements in data quality and curation have been the primary drivers of efficiency gains, rather than algorithmic or model architecture improvements. The study, which trained various model recipes with different data corpuses across multiple compute scales, found that data improvements accounted for a 12.0x increase in compute efficiency, while model improvements contributed only 3.7x. This suggests that much of the observed AI progress during this period, particularly in pretraining, was fueled by better data engineering, including more sophisticated filtering and larger dataset sizes. The findings have significant implications for the economic models of leading AI labs, potentially shifting focus and investment towards data acquisition and processing. While model innovations have been crucial for enabling larger-scale training by overcoming technical constraints, their direct contribution to compute efficiency appears secondary to data advancements for smaller models. The research also raises questions about future progress, specifically regarding the potential limitations imposed by data availability and the effectiveness of synthetic data in expanding corpuses.

AI Signal Decode

The core finding indicates that between 2019 and 2025, a 3.24x increase in pretraining compute efficiency at a 1e19 FLOPs budget was predominantly driven by data improvements (12.0x) compared to model enhancements (3.7x). This suggests that optimizing data collection, curation, and filtering techniques has yielded more significant returns in terms of achieving better performance for a given computational cost than advancements in model architectures or training algorithms alone. The additive nature of these improvements, with data and model contributions largely independent, means that gains from one do not necessitate specific conditions in the other.

Market implications center on the economics of AI development. If data quality and quantity are the primary levers for pretraining efficiency, then the cost and effort associated with data pipelines—including broader internet scrapes, advanced filtering mechanisms, and potentially synthetic data generation—become paramount. This could redirect significant investment away from pure algorithmic research towards data infrastructure and management. Furthermore, it raises concerns about a potential 'data wall' if the rate of new, high-quality data generation slows, which could stall pretraining progress.

Technically, while model innovations from GPT-2 to OLMo-2 have addressed critical scaling challenges like gradient stability and memory constraints, enabling larger and more complex models, their direct impact on *efficiency* in terms of FLOPs-to-performance has been outpaced by data improvements. This is particularly true for smaller models where data quality is highly sensitive to capacity. For very large models, the analysis suggests that their immense capacity might make them more resilient to lower-quality data, allowing them to extract signal from larger, less curated datasets, akin to a 'container ship' analogy for data ingestion.

Looking ahead, critical questions revolve around the scalability of data improvements and the role of synthetic data. The study did not investigate synthetic data's effectiveness, which is a major area of current AI lab investment. If synthetic data can meaningfully expand high-quality corpuses, it could alleviate the 'data wall' concern. Additionally, scaling this analysis to frontier model sizes and exploring the marginal value of novel, high-quality data in post-training phases (like RLHF) are crucial next steps to understand the true landscape of AI progress drivers.

Analysis

Covered by: techmeme