Q&A with AI researchers John Schulman, Beren Millidge, and Charlie O'Neill on steelmanning the case against RSI, Chinese labs' progress, long-horizon RL, more

The development of AGI might be significantly slower than predicted due to fundamental limitations in AI generalization and self-correction.

AI researchers John Schulman, Beren Millidge, and Charlie O’Neill discussed the potential limitations on achieving artificial general intelligence (AGI) and recursive self-improvement (RSI) in a recent podcast. They explored scenarios where AI development might plateau, citing the persistent "sim-to-real" gap as a potential long-term bottleneck, despite current rapid advancements. The researchers noted that even as models excel at specific tasks, they may continue to struggle with true generalization and self-correction, leading to recurring cycles of initial user amazement followed by the realization of limitations. Another significant point raised was the potential need for fundamental paradigm shifts beyond the current transformer and reinforcement learning (RL) architecture, suggesting that scaling existing methods might not be sufficient to overcome future hurdles in AI research and development, similar to how Moore's Law required discrete innovations. The discussion also touched upon the progress of Chinese AI labs and the role of RL in driving AI capabilities, questioning whether current approaches can lead to explosive growth or if they will eventually hit an asymptotic curve.

The researchers argue that current AI paradigms, primarily based on transformers and RL, might require paradigm-shifting innovations rather than mere scaling to achieve true AGI. They suggest that the "spark of generalization" could remain elusive, leading to AI systems that are highly proficient in benchmarked tasks but fail to achieve broad, human-like adaptability. This viewpoint implies that the current trajectory of AI progress, while impressive, may eventually encounter diminishing returns unless entirely new foundational concepts are discovered. The potential for hitting an asymptotic curve suggests that the rapid "takeoff" scenarios for AGI might be overly optimistic if such discontinuities are not found. Furthermore, the discussion highlights the challenge of models effectively "checking themselves" and the limitations imposed by judgment and error correction. If these issues are not resolved through new architectural breakthroughs, the cycle of impressive but ultimately limited AI models may continue indefinitely. This also raises questions about the future of AI research itself; if current models cannot independently discover novel paradigms, human ingenuity will remain the critical factor, potentially slowing the pace of AGI development considerably and challenging the very notion of self-improving AI systems.

]