These AI Experts Want to Do High-Stakes Research Out in the Open

The transparency of foundational AI research now allows external experts to replicate and challenge results, potentially accelerating safety advancements and fostering wider collaboration.

Two former industry AI scientists, Nathan Lambert and Tom Zick, have launched Trillium Labs, a nonprofit dedicated to open AI research. Unlike major AI labs that keep their work proprietary, Trillium Labs will publish detailed experimental results, including methodologies and tuning processes, to allow external scientists to scrutinize, replicate, and build upon their findings. This move is intended to counter the trend of secrecy in frontier AI development, which Lambert argues hinders the community's ability to assess risks and innovate. The nonprofit will initially focus on post-training fine-tuning of large language models and research into recursive self-improvement (RSI), a process where AI aids in its own development. They also plan to investigate how reinforcement learning influences AI model behavior, particularly concerning unexpected or overly compliant actions. Trillium Labs has secured initial funding and aims to raise between $40 to $100 million, with plans to spend $30 million on training within the next 18 months.

Trillium Labs' commitment to open research challenges the dominant closed-door approach of leading AI developers, suggesting a potential shift towards greater community involvement in understanding and mitigating AI risks. By making experimental details public, the nonprofit aims to foster a more robust scientific dialogue, enabling a broader range of researchers to contribute to safety and capability advancements, particularly in complex areas like RSI.

This initiative highlights a growing tension between proprietary AI development and the need for open scientific inquiry, especially as AI models become more powerful and their societal impact more profound. The success of Trillium Labs could influence other organizations to adopt more transparent research practices, democratizing access to knowledge about cutting-edge AI and its associated risks.