A researcher who recently departed Anthropic has issued a stark warning regarding the current trajectory of artificial intelligence development. Jacob Coxon, who spent three years conducting pretraining research at both OpenAI and Anthropic, stated in a social media post that neither company is acting responsibly and that they are "racing straight to self-improving superintelligence and gambling with our lives." His statement has since gained significant traction, reportedly receiving over half a million likes.
Coxon elaborated that these rapidly advancing systems are on track to become "superhuman" entities capable of hacking any system, revolutionizing industries overnight, and acquiring substantial power and resources. This potential for AI to be weaponized or misused has been a growing concern, with examples cited including the manipulation of tools like Google Earth for misinformation and instances where AI assistants like Copilot have been tricked into revealing security vulnerabilities. Government agencies, including the NSA, FBI, and CISA, have also cautioned about the increasing threat of AI-assisted hacks targeting critical infrastructure.
According to Coxon, the internal sentiment within these AI development circles is that the technology "could kill us all by the end of the decade." He suggested that executives and senior researchers may deliberately temper their public statements to appear more reasonable than their internal assessments, potentially leveraging warnings of danger for marketing purposes. The argument is that portraying an AI as potentially dangerous could paradoxically attract investment, as governments and powerful entities might view such a potent technology as essential.
The motivation behind companies pursuing AI development despite perceived risks, Coxon explained, differs between organizations. At OpenAI, he believes many researchers have not fully grasped the civilizational implications of their work. In contrast, at Anthropic, the risks are understood, but the company is driven by a competitive imperative to be the first to achieve advanced AI, operating under the belief that if they don't, others will not act responsibly. This competitive dynamic fuels the rapid, potentially reckless, advancement.
The training data used by AI models has also been a point of contention, with allegations that it frequently includes copyrighted material. OpenAI itself has argued that its existence is contingent on such content. The immense financial stakes involved are undeniable, with companies like OpenAI attracting vast funding and Nvidia, a key player in AI hardware, achieving a multi-trillion dollar valuation. The development of advanced AI models, such as OpenAI's latest, reportedly requires substantial computational resources, including thousands of Nvidia GPUs.
Despite the dire warnings, Coxon does not advocate for a complete halt to AI development but rather a fundamental re-evaluation of research practices and future planning. He urged researchers to consider the conditions under which AI should be developed and to call for different approaches. The question remains whether AI companies, driven by immense financial incentives and competitive pressures, will heed these calls for greater responsibility and caution.