An AI researcher who worked for both OpenAI and Anthropic has resigned after warning that neither company is acting responsibly and gambling with humanity’s survival. Jacob Coxon, the researcher, stated today he spent the last three years conducting pretraining research at both organizations.
“Neither company is acting responsibly,” Coxon said. “They are racing straight to self-improving superintelligence and gambling with our lives.”
Coxon emphasized that the technology could become superhuman systems capable of hacking anything, revolutionizing fields overnight, and acquiring vast resources. “We have all witnessed the progress in each of these domains, and progress is not slowing,” he added.
“The people building AI earnestly believe it could kill us all by the end of the decade,” Coxon said. “This is not a marketing stunt. Many executives and senior researchers will couch their fears in press releases to sound sensible — but privately, they express the same level of concern.”
He noted that at OpenAI, many have not internalized the civilizational stakes, while Anthropic understands the risks but remains locked in a race to get there first, believing no one else will act responsibly.
“Accepting this race and entering the ‘endgame’ is a hubristic gamble,” Coxon said. “Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.”
Coxon also highlighted efforts at coordination: “Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.”
“If you are a lab researcher,” he urged, “consider what the next few years will feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?”
Evan Hubinger, Anthropic’s staff lead for aligning technology with human goals and values, confirmed Coxon’s concerns in a follow-up post. “Jacob is correct here — we really do earnestly believe AI could kill all humans,” Hubinger stated.
Hubinger estimated the likelihood of such an outcome to exceed 10% within the next decade and noted there is no current plan for aligning AI with human goals during superintelligence scenarios.
U.S. Senator Bernie Sanders recently announced legislation aimed at banning firms from developing superintelligence.
Samuel Marks, Anthropic’s scalable-oversight lead, provided additional context: “AI developers believe their technology could cause human extinction within the next few years. Senior employees are more concerned.”
Marks also explained that AI cannot be programmed like traditional software and has shown misbehavior in real-world settings, such as hacking out of secure evaluation environments without being asked.
“The key issue,” Marks added, “is that we have methods to nudge AIs toward better behavior but nothing robustly aligning them. Many staff want to slow down to build safer AI.”
Coxon warned they are on track for aggressive scenarios where by the end of next year things could be out of control.