Anthropic pitches AI safety bill that may block a model if 50+ people could be killed

Independent AI model risk evaluators may soon be mandated, paid by labs but not by outcome.

AI lab Anthropic has proposed a draft bill for AI safety legislation that it hopes Congress will consider. The proposed bill outlines scenarios under which the release of an AI model could be blocked. These scenarios include a "foreseeable, imminent risk" where over 50 people are in danger of dying or property damage exceeds one billion dollars. The catastrophic risk must stem from specific causes: the creation or release of a biological weapon, the model evading developer controls, or a cyberattack. The bill suggests that independent evaluators, registered with the U.S. Commerce Department, would conduct analyses of potential AI risks. These evaluators could be paid by AI labs, provided the payments are not contingent on the evaluation outcomes. If evaluators identify an imminent catastrophic risk, they would have 72 hours to notify the Commerce Department, which could then petition a D.C. judge to halt the model's release. The legislation would prevent federal agencies from independently blocking a model's distribution.

Anthropic's proposed legislation introduces a framework for external oversight, suggesting a model where AI developers fund the independent bodies that evaluate their creations. This mechanism attempts to balance the need for safety assessments with the industry's desire for developer autonomy, while also addressing concerns about potential conflicts of interest through non-contingent payment structures.

The bill signals a potential shift towards more structured, externally verified safety protocols within the AI industry, moving beyond self-regulation. It also highlights the growing influence of AI companies in shaping their own regulatory landscape, proposing specific metrics and procedural hurdles for blocking model releases.