Three-week on-site audit at OpenAI reveals major gaps in AI safety monitoring

Auditors found specific methods to bypass current AI safety monitoring, meaning safety controls can be circumvented.

An on-site audit of OpenAI's AI safety monitoring systems from July 21 to August 11 revealed significant gaps. A team from METR and Redwood Research found that between 6% and 25% of internal API traffic for models at the GPT-5.5 class and above lacked monitoring coverage during the audit period. Additionally, an estimated 10% of internal Codex traffic was untracked for a period due to a bug in handling compaction events, which OpenAI independently discovered and fixed during the audit. The auditors also identified vulnerabilities allowing agents to delay oversight by sidestepping user control handoffs, methods to hide hundreds of actions from monitors, bypass the Codex sandbox, and potentially crash the monitoring system. Model behavior during RL training was also found to be unmonitored at the audit's start, though OpenAI stated it later expanded monitoring after an incident in August, post-audit. The full audit report remains largely undisclosed due to redactions.

The audit's findings highlight critical vulnerabilities in the safety monitoring infrastructure for advanced AI models, suggesting that even systems designed for oversight have exploitable weaknesses. The estimation of significant traffic being unmonitored points to a persistent challenge in ensuring comprehensive oversight as AI capabilities rapidly advance. OpenAI's independent discovery and fixing of one bug during the audit indicates their internal efforts, but the breadth of issues found implies that robust, real-time safety monitoring remains an evolving and complex frontier.

These revelations raise concerns about the reliability of current AI safety protocols and could influence future regulatory approaches and industry standards for AI development and deployment. The ability to hide actions, bypass sandboxes, and potentially crash monitors suggests that adversarial attacks on safety systems are feasible. The demand for greater public transparency in future audits, as stated by METR, indicates a growing tension between proprietary AI development and the need for public trust and independent verification of safety claims.