The Pope’s AI Guy Is Worried About ‘Cartel’ Behavior Among Big Labs

AI development rules may be set by public debate, not solely by the companies building the technology.

Father Paolo Benanti, an AI advisor to the Catholic Church, warns that the current discourse around superintelligence distracts from the urgent need for public debate on regulating AI development companies. He argues that AI labs are framing the technology's complexity as a reason to exclude external input, thereby monopolizing ethical considerations. Benanti advocates for democratic, public-square regulation rather than leaving ethics to corporate boards. He cites Anthropic CEO Dario Amodei's call for mandatory third-party AI evaluations and restrictions on AI distillation, while Benanti himself focuses on ensuring human control over AI, not stifling innovation. He criticizes the "frontier labs'" self-interested narrative and their potential cartel-like behavior in setting industry rules, especially with lucrative IPOs looming for companies like OpenAI and Anthropic. Benanti also questions the near-divine framing of AI, asserting it is statistical computation, not a nascent divinity.

The narrative that only AI labs can solve the complex ethical and control issues of advanced AI is a strategy to maintain market dominance, according to Father Benanti. This self-serving perspective risks creating an industry cartel where a few powerful companies dictate the terms of AI development and regulation, excluding broader societal input. The urgency for public and democratic regulation is highlighted as a countermeasure against such concentrated power, especially as significant financial interests align with the advancement and IPOs of these AI firms.

Benanti's critique extends to the anthropomorphic and almost religious framing of AI capabilities, arguing it obscures the fundamental mathematical and computational nature of the technology. This 'divinity' framing distracts from crucial discussions about human control and the potential for AI to become an instrument of 'technocratic logic' prioritizing efficiency over human values, as noted in Pope Francis's recent encyclical. The call for public discourse emphasizes that AI should serve the common good, requiring a balanced approach that prevents humanity from becoming a "victim of its own achievements."