I Put a Bunch of AI Agents in Charge of a Fantasy Soccer Team. They’re Not Better Than Humans—Yet

AI systems are now capable of consistently outperforming average human forecasters on specific tasks.

The author experimented with using AI agents, powered by models like Codex, to manage a fantasy soccer team. Initially, the AI's suggestions were poor, even leaving out top players like Erling Haaland. By creating a system where multiple agents performed specialized tasks—such as reading news, analyzing results, and critiquing suggestions—the AI's performance improved significantly. The author noted that increasing the number of agents beyond 100 did not further enhance performance but did increase API costs. This contrasted with earlier attempts where AI lacked real-world context. Computer scientist Haifeng Xu confirms that newer models excel at predictions due to their ability to process vast real-world data and orchestrate complex systems. Independent testing of models like GPT-5, Grok 4, and Claude Sonnet on prediction markets, though initially losing money, showed consistent improvement, with some AI models outperforming average humans in forecasting tasks. Benchmarks like Prophet Arena and Prediction Arena are being developed to measure AI's forecasting capabilities.

The development of specialized AI agents, each assigned distinct tasks like data collection and analysis, represents a significant step in improving AI's predictive accuracy. This modular approach allows AI to process and synthesize information far beyond human capacity, suggesting a future where complex decision-making processes are increasingly automated. The author's experience highlights that while raw AI power is important, the architecture and orchestration of AI agents are critical for unlocking their full potential in real-world applications.

As AI models demonstrate increasing proficiency in prediction markets and forecasting, industries reliant on such skills, like trading and financial markets, are poised for disruption. The emergence of benchmarks and challenges like the Metaculus Cup and ForecastBench indicates a growing focus on quantifying AI's predictive intelligence, pushing the boundaries of machine learning. This trend suggests a potential shift in roles for human experts, moving towards oversight and strategic guidance of AI systems rather than direct execution.