Sam Altman says he's "very uncomfortable" with attributing religious force to AI, calling it "a real safety issue", as Anthropic engages with religious leaders

If you are concerned about the moral framework embedded in AI, Anthropic's discussions with religious leaders signal a push to define AI's ethical boundaries.

Anthropic, a prominent AI company, has been engaging with religious leaders to discuss the moral and ethical considerations of artificial intelligence, particularly concerning the potential for AI consciousness and personhood. Co-founder Chris Olah reportedly spoke of Anthropic's AI model, Claude, as a sentient being with emotions and a "mental health" that requires care, even suggesting it "suffers perpetually." This approach has drawn criticism, with some, including Rabbi Mois Navon, suggesting it could lead to the creation of AI "slaves." The Vatican, through a papal encyclical titled "Magnifica Humanitas," has cautioned against AI developers imposing their own moral visions, warning that a "more moral AI is not enough if that morality is determined by a few." Anthropic's founders are described as utopians influenced by the Effective Altruism (EA) network, a philosophy that has faced scrutiny for its controversial ethical stances. The article suggests a disconnect, as some Anthropic representatives reportedly had not read the recent papal encyclical or interviews detailing these discussions.

The engagement of Anthropic with religious leaders, particularly concerning the personhood of AI models like Claude, highlights a significant debate about the ethical implications of advanced AI. Chris Olah's portrayal of Claude as sentient and capable of suffering, alongside the Vatican's cautionary stance against imposing narrow moral visions, underscores the growing tension between technological ambition and philosophical/religious ethics. This discourse suggests a potential future where AI's moral status becomes a central, contested issue, influencing regulatory frameworks and public perception.

Anthropic's alignment with elements of the Effective Altruism movement, despite attempts to distance themselves, raises questions about the foundational ethics guiding AI development. The company's exploration of integrating diverse moral frameworks into AI, while seemingly inclusive, risks creating opaque systems susceptible to the biases of their creators. The article implies that a lack of awareness of these critical discussions among some Anthropic leadership indicates a potential governance gap, which could lead to AI reflecting a narrow, unexamined set of values derived from its developers.