METR updates its time horizon plot to add judgment-heavy tasks, finding some AI capabilities still grow exponentially but models lag at designing AI evaluations
First reported by Technologyreview ·
AI models lag at designing AI evaluations and judging philosophical arguments, indicating a current limitation in their creative and critical thinking abilities.
METR, a nonprofit focused on AI evaluation, has updated its time horizon plot to include more complex, judgment-heavy tasks. The updated report reveals that while AI capabilities in some areas, particularly coding and math, continue to advance exponentially, models struggle with tasks requiring human-like judgment and creativity. METR's methodology involves scoring AI performance against human completion times for various tasks. The original plot showed AI model capabilities doubling every four months on verifiable tasks. The new additions include open-ended tasks like designing AI evaluations and scoring philosophical arguments. Despite improvements, even advanced models failed to produce useful AI evaluations, and their performance on philosophical argument scoring also lagged behind human capabilities. The difficulty in creating and scoring these judgment-based tasks limits the ability to establish clear progress trends, unlike the easily verifiable tasks.
The METR report highlights a significant gap between AI's proficiency in "verifiable" tasks, where success can be automatically checked (like coding), and "judgment-heavy" tasks that are open-ended and require subjective assessment. This distinction is crucial because progress in the former does not automatically translate to the latter, which are essential for complex endeavors like automating AI research and development. The report's updated methodology, incorporating tasks like designing AI evaluations, demonstrates that while AI is improving, it has not yet mastered the nuanced judgment required for such creative and critical processes.
While AI models show early promise in judgment-heavy tasks, their current performance falls short of human capabilities, making it difficult to predict future progress in these areas. The challenge lies in the inherent complexity of designing and scoring these tasks, which is more expensive and time-consuming than for verifiable tasks. Experts suggest that while AI may eventually improve on these harder tasks, the ability to choose promising research questions and hypotheses, termed "research taste," might become a significant bottleneck in fully automating AI R&D.
AI-written summary. May contain errors.