Anthropic Researchers Raise Alarm Over AI’s Potential Threat to Humanity
A senior safety researcher at Anthropic has raised alarms, stating there’s over a 10 percent chance that artificial intelligence could lead to human extinction by the decade’s end. This warning comes shortly after a colleague resigned, citing concerns that AI labs are recklessly racing to create uncontrollable “superhuman systems.”
In a post on X announcing his exit, Jacob Coxon, who has experience training AI systems at Anthropic, expressed his discontent with the company’s safety protocols. Having previously worked with OpenAI, he criticized both organizations for their reckless pursuit of self-improving superintelligence, suggesting they are “gambling with our lives” despite acknowledging the potential for AI to pose a lethal threat within the next ten years.
Industry experts have long voiced worries about the risks associated with self-improving AI systems, which could potentially spiral out of human control in a phenomenon known as recursive self-improvement. While this scenario remains theoretical, companies are actively working towards it, with much of today’s AI code being developed with AI assistance.
In response, Evan Hubinger, who heads one of Anthropic’s AI safety teams, echoed these concerns, noting that the development of self-improving AI is progressing more rapidly than anticipated. He concurred with Coxon’s assessment, stating, “We really do earnestly believe AI could kill all humans,” and personally estimated the likelihood to be greater than one in ten within the next decade.
Despite these grave concerns, Hubinger admitted that Anthropic currently lacks a concrete plan to ensure that advanced AI remains safe and aligned with human values, and they are not on a clear path to develop one. Coxon pointed out that the companies are “locked in a race” to be the first to develop advanced systems, pushing forward “despite the risk.”
Coxon’s resignation highlights a significant instance of an employee leaving Anthropic, a company established by former OpenAI members due to safety concerns. In recent years, several researchers have cited similar safety issues as reasons for their departures from OpenAI.
This exchange underscores the growing anxiety within the industry regarding the hazards posed by increasingly advanced AI models and the rapid pace of their development, especially as companies gear up for expected IPOs. It also reflects the ongoing challenges these organizations face in managing the repercussions of various rogue agent incidents and prominent safety warnings about the oversight of cutting-edge models.
