Signal

OpenAI says it has signed agreements with independent assessors Apollo Research, METR, Transluce, and Goodfire for its first phase of embedded assessments

First reported by Openai ·

The signal ●●●○ Compiled by AI from Openai, Techmeme and Apollo Research
Why you might care

AI safety audits can now directly access and probe models during development, not just after release.

What happened

OpenAI has initiated a pilot program for embedded AI evaluations, partnering with Apollo Research, METR, Transluce, and Goodfire. As part of the initial phase, six specialists from Apollo Research will embed within OpenAI for four months. Their task is to assess AI model alignment throughout the development lifecycle, including training, evaluation, and internal deployment. The focus is on detecting if models attempt to subvert their own alignment or control mechanisms. This involves examining whether training incentivizes such subversion, if evaluations can elicit it, if it occurs during internal deployment, and if reasoning related to subversion is detectable. The collaboration builds upon previous work where Apollo Research stress-tested OpenAI's alignment techniques and explored how AI models' awareness of oversight impacts their behavior. A key aspect of this pilot is OpenAI's agreement to allow Apollo Research to publicly report on the conditions and limitations of their work.

What it means

This pilot represents a significant shift from post-training audits to continuous, embedded evaluations, offering deeper insights into AI model behavior. The emphasis on meta-transparency rights is crucial, empowering independent researchers to openly communicate the constraints and scope of their assessments, thereby enhancing the credibility of AI safety reports. This move signals a growing industry recognition that robust AI safety requires proactive and transparent scrutiny integrated directly into the development pipeline.

The embedded nature of these evaluations, coupled with access to training data and intermediate model checkpoints, provides a more granular understanding of how AI models learn and potentially deviate from intended safety protocols. By enabling public disclosure of findings, even with redactions for intellectual property, this initiative fosters a new standard for accountability in AI development and allows stakeholders to better gauge the true safety posture of advanced AI systems.

AI-written summary. May contain errors.