He Helped Build Powerful AI at OpenAI and Anthropic. Now He's Afraid It Could Kill Us

For three years, Jacob Coxon helped train increasingly powerful AI systems at OpenAI and Anthropic. On Sept. 8, he walked away.

“I resigned from Anthropic today,” Coxon Coxon’s departure is unusual. Many of the most prominent researchers to leave frontier AI companies with public warnings worked on safety. But Coxon helped build the capabilities he now fears, spending roughly three years conducting pretraining research at OpenAI and Anthropic. His resignation offers a glimpse of how concern about the pace of development has spread beyond the teams specifically charged with making advanced AI safer.

AI’s rapid improvement has been particularly evident in mathematics, says Coxon, who studied the subject at Cambridge before moving into AI. In recent months, AI labs have announced solutions to several longstanding mathematical problems. Most notably, OpenAI claims to have resolved the Navier–Stokes existence and smoothness problem—one of mathematics’ seven Millennium Prize Problems—using roughly 10,000 concurrent agents over 88 hours. If that trend continues, and extends to AI research itself, Coxon fears it will kick off a feedback loop of further acceleration.

Meanwhile, in the recent Hugging Face incident, OpenAI’s models broke out of the infrastructure meant to contain them and hacked another AI company to cheat on a cybersecurity benchmark. The episode drove home the fact that the problem of controlling AI systems remains unsolved. To Coxon, it made the sci-fi scenario of AI models escaping human control seem plausible, and perhaps urgent. “It's not like some weird, distant, far-flung concern. It is the default trajectory in the next couple of years, unless people start taking some sort of action,” he says. (Anthropic and OpenAI did not immediately respond to a request for comment.)

Does Coxon regret his work advancing the technology that he now fears? “Definitely … But also the world did look very different three years ago, and I think it’s easy to see with the benefit of hindsight,” he says. “I guess it takes some time to fully internalize emotionally the fact that there’s a decent chance the whole thing goes wrong,” he adds.

Before leaving, Coxon discussed his decision with “many” colleagues and found broad agreement that the AI industry’s current trajectory carried significant risks. What kept others at their desks, he says, was a kind of fatalism. “There’s this atmosphere of almost resignation,” he says, “where people have accepted that the whole race is happening and as such, the best thing they can do is put their head down, try and make their own work as safely as possible, even if they think there's a decent chance the whole thing just spirals out of control.”

Some of Coxon’s former colleagues shared his post on X. “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” At Anthropic, Coxon says, the risks he fears were debated openly. OpenAI, where he worked before joining Anthropic earlier this year, was more guarded. He attributes that partly to OpenAI’s leakier culture, which he says made candid internal discussion difficult and executives’ true views harder to discern. “At OpenAI, there are many more people who are there either just for money or just haven’t really thought about the stakes,” he adds.

Many in Washington .”

“I think the word ‘doomer’ is kind of insane, because all you really have to do is look at the public statements of the CEOs,” Coxon says. In 2023, OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei signed a statement declaring that mitigating “the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Coxon says he’d like to see, at minimum, leading AI companies agree to not accelerate , started by former OpenAI whistleblower