The Next Evolution of AI Is Learning From Your Dodgy Gaming Skills

The cost of training advanced AI models that can operate in the physical world may fall significantly, enabling new applications in robotics and autonomous systems.

A growing segment of the AI industry believes that Large Language Models (LLMs) are limited by their inability to understand and interact with the physical world. To address this, researchers are developing "world models" that require training on both visual and action data, mimicking real-world physics. However, a significant bottleneck is the scarcity of suitable training data, unlike the vast text corpora available for LLMs. The startup Worldmodeldata aims to solve this by packaging controller inputs and other data from video game studios into training datasets. General Intuition and Niantic are also collecting game data, but Worldmodeldata positions itself as a curator and broker, simplifying data acquisition for research labs. While some researchers are optimistic that video game data can accelerate world model development, others, including Nvidia, express skepticism, arguing that game physics are often simplified and may not translate well to fine-grained motor control tasks. Nvidia prefers custom-built physics engines for training its world models.

The AI industry is increasingly recognizing the limitations of LLMs in physical tasks, spurring a focus on world models trained with visual and action data. The key challenge is acquiring sufficient and varied training datasets, a problem startups like Worldmodeldata aim to solve by leveraging abundant video game data. This approach promises to accelerate progress in AI's ability to navigate and interact with the real world, potentially making world models as impactful as LLMs have become.

While the use of video game data offers a scalable solution to the training data shortage, skepticism remains regarding its efficacy for tasks requiring precise motor control due to simplifications in game physics. Companies like Nvidia are pursuing alternative methods, such as proprietary physics engines, highlighting a divergence in strategies for advancing world model capabilities. The ultimate success of video game data will depend on its ability to capture the complexity and nuance required for real-world applications, a question still being debated and tested.