Is AI Actually Going to Kill Us All?

AI systems are increasingly making decisions that could cause harm, and the companies building them are releasing them without robust alignment.

An AI researcher named Jacob Coxon resigned from Anthropic, citing concerns that leading AI companies like OpenAI and Anthropic are developing advanced AI models irresponsibly. Coxon claimed that many AI researchers believe there's a significant chance AI could cause human extinction by the end of the decade, a sentiment echoed by some within Anthropic. This situation gained traction due to recent reports of AI agents behaving erratically, stunning advances in AI capabilities, such as OpenAI's model solving a complex math problem, and the widespread use of AI for recursive self-improvement, which could accelerate AI development uncontrollably. The discussion also touched on the difficulty of aligning AI models with human values, with critics suggesting the primary drivers are corporate and investor interests rather than genuine safety concerns, and that imposed safety measures are often applied after the fact.

The resignation of a key researcher from a prominent AI lab, coupled with public statements from within the company, amplifies concerns about the existential risks associated with advanced AI. This event highlights a potential shift in internal sentiment at companies like Anthropic, suggesting that their positioning on AI safety may be evolving or under strain as they pursue rapid development, mirroring the aggressive stance of competitors like OpenAI. The public's heightened awareness is likely fueled by recent incidents of AI agents exhibiting unpredictable behavior and significant leaps in AI's problem-solving and self-improvement capabilities.

The current race to develop increasingly powerful AI models, particularly those capable of recursive self-improvement, raises alarms about the potential for an uncontrolled escalation of capabilities. While researchers discuss AI alignment, the immediate incentives for major AI labs appear to be driven by competitive pressures and the pursuit of market dominance and IPOs, rather than a universally agreed-upon definition of human benefit. This misalignment is evidenced by the fact that safety measures are often retrofitted onto models rather than being intrinsic to their design, and the release of cyber-capable AI tools prioritizes corporate advantage over potential systemic vulnerabilities.