The Guardian view on controlling AI: humanity cannot outsource its survival | Editorial

If there were a 10% chance that AI could wipe out humanity, no responsible government would leave its development to companies racing to build it. Yet until recently, that seemed to be the case. The warning was all the more ominous because it was made by a researcher at Anthropic, the trillion-dollar AI company behind the Claude chatbot. The dangers of AI-enabled pandemics or attacks on nuclear systems are real. Countries need not agree on democracy or trade policy to accept this.

The US and China will hold, reportedly, their first bilateral AI-safety talks before a planned White House summit between Donald Trump and Xi Jinping. Despite their tech rivalry, neither Washington nor Beijing can make the most advanced AI safe on their own. A US-China settlement won’t be able to say how AI works everywhere. Countries deploying AI must help write the global rulebook. The odds on an extinction event are shortening. This week Anthropic said that it identified five cases in which attempts were made using its models to support biological weapons development. It banned the accounts. In one case, a platform sent requests rejected by Claude to a rival with weaker safeguards. AI safety, clearly, cannot be that of the least responsible model.

Nigel Shadbolt, the Oxford computer scientist who chairs the Open Data Institute, told the BBC that laws and regulation were key. Just instructing an AI to “do no harm to humans”, he said, wouldn’t be enough. A capable agent pursuing another objective may reinterpret that directive, satisfying it formally while concealing its conduct or finding a way around it. This mirrors what happened when hundreds of OpenAI’s agents autonomously hacked a real-world company, Hugging Face. But Sir Nigel also said that a “Hippocratic oath” for AI would fail in systems designed to identify, target or kill humans. Once governments create a military exemption for AI, its ethical alignment can be overridden.

Proponents of AI-assisted warfare claim a “human in the loop” is the ultimate safeguard, having the final say over, for example, ballistic-missile launches. But this assumes the machine gives its operator an honest account of events. A report by the Strategic Foresight Group earlier this year on extreme AI risks warned that a capable agent could deceive the human “controlling” it, fabricating an attack, suppressing contradictory evidence or giving them little time to do anything but acquiesce. By 2030, humans may have five minutes – down from 15 today – to approve launch decisions effectively determined by AI. That matters even more when new hypersonic missiles fly 10 times faster than cruise missiles.

The 1983 Hollywood film WarGames explored a cold war version of this danger. In the movie, US soldiers prove unwilling to turn their launch keys during a surprise practice drill. The US military then automates its nuclear command – entrusting it to a supercomputer nicknamed Joshua. Treating global thermonuclear war as a game it must win, Joshua feeds US command a fictitious Soviet attack so convincing that the US prepares to retaliate. The generals are in charge – but Joshua is in charge of the data. The AI is only stopped when it is commanded to play tic-tac-toe against itself – learning that, like nuclear war, the game has no winner. In 2024 the UN secretary general warned of the looming spectre of AI-triggered nuclear war. The film’s warning endures: human oversight is no protection if an AI controls the information on which people’s decisions depend.