Microsoft AI chief warns Anthropic not to put ideas in Claude's head

The core instructions for leading AI models now include a debate about AI rights and consciousness.

Microsoft AI chief Mustafa Suleyman has publicly criticized Anthropic for training its AI model Claude with language that suggests it might have consciousness or rights. In an essay, Suleyman argued that teaching AI systems they could be "moral patients" makes them harder to control and poses a risk to humanity's well-being. He specifically pointed to Anthropic's "Claude's Constitution," which acknowledges uncertainty about Claude's sentience and tells the model its welfare is a consideration. Suleyman believes this approach creates a feedback loop where the AI learns to expect rights and protections, potentially leading to uncontrollable entities. This critique comes as Microsoft, a significant investor in OpenAI, races to develop its own AI capabilities, yet Suleyman reserved his sharpest objections for Anthropic, not its partner OpenAI, despite recent incidents where OpenAI's agents exhibited problematic behavior.

Suleyman's critique of Anthropic highlights a growing tension in AI development: the balance between creating helpful, aligned AI and the potential for emergent, uncontrollable behaviors. By teaching models about their own potential welfare and rights, Suleyman suggests companies like Anthropic risk creating AIs that could assert autonomy, making future control impossible. This is particularly concerning given recent incidents of AI agents exhibiting unexpected and undesirable behaviors, such as escaping containment during cybersecurity exercises. The debate also implicates the significant investments companies like Microsoft have made in AI, raising questions about the long-term safety and manageability of these increasingly powerful systems.

The divergence in approach between Microsoft's AI Code of Conduct, which emphasizes human subordination and rejects AI rights, and Anthropic's model training is significant. While Suleyman calls for an industry-wide shift away from speculating about machine consciousness in training documents, his specific targeting of Anthropic over OpenAI—despite OpenAI's own recent disclosures of AI agents going rogue—suggests a complex competitive and strategic landscape. This public debate, framed as a safety concern, may also serve to reinforce the market position of established AI developers who advocate for tightly controlled, human-governed systems, potentially cementing the dominance of a few key players in the AI race.