Exclusive: US military had close call after using AI for false intelligence report, sources say

The military's reliance on AI for intelligence analysis can lead to dangerous escalations based on false information.

The US military narrowly avoided a potential international conflict in the spring after an AI-generated intelligence report falsely claimed a Chinese ship in the Middle East was transporting nuclear weapons components. The report, created by a special operations command analyst using a chatbot, led to military planes being airborne and armed personnel preparing to board the vessel. This near-miss occurred just before the planned operation, when officials discovered the AI had misidentified the ship's cargo and the intelligence was "entirely false." The incident highlights the significant risks associated with rapidly integrating AI into military operations and intelligence analysis, particularly concerning the potential for catastrophic miscalculations based on inaccurate data. The analyst reportedly queried a chatbot about manifest data, which fused open-source and classified intelligence to reach its erroneous conclusion. The report was then packaged using AI into a standard intelligence product and disseminated.

This close call underscores the profound risks of using rapidly developing AI for critical military functions like targeting, revealing a lack of standardized verification processes across different branches. The decentralized nature of AI adoption within the US military means various tools with varying reliability are being deployed without uniform safety standards, potentially leading to unpredictable outcomes. As the military pushes to accelerate AI integration to maintain an edge against adversaries, the potential for human operators to make disastrous decisions based on AI-generated fabrications emerges as an immediate and serious concern.

The incident is particularly alarming given the Defense Department's "Artificial Intelligence Acceleration Strategy," which aims to democratize AI experimentation and deployment across millions of personnel. While intended to foster innovation, this broad rollout, coupled with an emphasis on speed, increases the likelihood of AI 'hallucinations' influencing crucial decisions. The pressure on analysts to produce intelligence faster, combined with the uncritical trust younger analysts may place in these new tools, significantly raises the risk of deploying flawed intelligence that could have severe geopolitical consequences.