This spring, amid the war with Iran, a U.S. Special Operations Command analyst reportedly used a chatbot to combine open-source material with classified signals intelligence about a Chinese vessel moving through the Middle East. The resulting assessment said the ship carried components for a nuclear weapons program.

The problem, according to accounts of the episode, was that the chatbot had misidentified the cargo manifest. The analyst then used the tool again to turn the finding into an official-looking report, allowing the error to move through command channels.¹

U.S. forces were preparing to intercept and board the vessel, with aircraft already airborne and armed personnel ready. Officials halted the operation only after taking a closer look at the underlying intelligence and finding the report was “entirely false.” One source’s assessment of the stakes was blunt: the failure “almost started a war.”²

The episode lands as the Pentagon pushes generative AI deeper into military work, arguing that faster analysis can sharpen decision-making. But the near-miss illustrates the opposing danger: an authoritative-looking AI output can acquire momentum before a human tests whether it is true. A separate Bloomberg-reported Pentagon inquiry into a deadly February 28 missile strike on an Iranian school also identified “overreliance on artificial-intelligence technology” among its failures.³

For Jake Steckler, a GovAI researcher and Army veteran, the lesson is not to abandon the technology but to constrain it. “It’s especially critical” to understand the uncertainty of large language models when decisions involve targeting, intelligence analysis or operational planning, he said, because those choices carry “life and death consequences.” He warned that prioritising adoption speed over safeguards could erode service members’ trust in the systems.¹

That is the central tension exposed by the aborted operation: AI may accelerate the military’s chain of command, but only disciplined human scrutiny can stop a fabricated conclusion from accelerating toward force.