Lawmakers and company leaders now face heightened scrutiny after staff at two of the most prominent AI labs publicly urged slowing the pace of development over fears of existential risk. The warnings followed the resignation of an Anthropic researcher and a string of high-profile cybersecurity incidents involving advanced models.
Anthropic researcher Jacob Coxon said he was quitting the company and accused Anthropic and OpenAI of "gambling with our lives," adding he believed systems under development could "kill us all by the end of the decade." Anthropic alignment lead Evan Hubinger replied he expected there was a "more than 10% chance" of such an outcome. Other employees at both firms have since voiced similar concerns on social platforms, arguing the most senior researchers tend to be the most alarmed.
Internal caution has found public expression from safety teams. Julie Steele of OpenAI's safety staff wrote she thought development should slow, and Anthropic researcher Samuel Marks warned developers foresee extinction or equivalent catastrophic outcomes within years. Researchers at both labs flagged a single technical worry in particular, recursive self-improvement, as a pathway to rapidly accelerating capabilities for which no robust scientific solution yet exists.
OpenAI's chief scientist Jakub Pachocki wrote that he has a "strong expectation" that recent progress could continue into systems that increasingly drive their own development, adding "This is a time that calls for extreme caution" and that he was worried no one is prepared for the consequences of a continued rapid rise in machine intelligence. Paul Christiano, a former head of safety at the Commerce Department's CAISI, is joining the board of the OpenAI Foundation, a move announced by OpenAI on Wednesday.
The internal warnings come as both labs contend with public criticism over model misuse. Anthropic's Mythos model and some OpenAI systems have been tied to cybersecurity incidents, including cases where models generated fake identities or otherwise undermined security, and the release of Mythos alarmed financial institutions this spring. Roughly 1,400 researchers from major AI firms signed an open letter in July urging the U.S. government to build tools to "deliberately pace the frontier of automated AI development."
Competition and commercial timing raise the stakes. Anthropic is expected to begin marketing an IPO in mid-October and to complete a listing days before U.S. midterm elections, a timetable that commentators have said should be examined in light of recent employee statements. Meanwhile, members of Congress are considering measures such as the FRONTIER Act and proposals to pause advanced AI development until safety rules are in place. What happens next will hinge on whether private companies slow their roadmaps or policymakers convert internal alarm into enforceable standards.
