Jacob Coxon published the viral post from a park bench in San Francisco’s Alamo Square. “I resigned from Anthropic today,” he wrote on Sept. 8. “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”
Coxon, a 27-year-old British researcher, hit send on X with no expectations. Instead, he became the unlikely catalyst of a chain reaction that changed the debate over artificial intelligence. “We really do earnestly believe AI could kill all humans!” added Evan Hubinger, who leads a department at Anthropic focused on ensuring AI acts the way its creators intend. “Unless there is AI regulation or a coordinated slowdown between labs, human extinction in the next few years seems very likely,” agreed Marcus Williams, who monitors AI agents at OpenAI, and pegged the risk at 70% without those steps.
Within 36 hours, Coxon’s original posts received 153 million views. Dozens of politicians joined the chorus of concern. On Sept. 12, Anthropic CEO Dario Amodei published a 3,800-word essay arguing that the mounting risks of AI warranted a slowdown. In as little as six to 12 months, he wrote, misaligned AIs “could be capable of taking over the entire internet.” Two of his fiercest rivals, Elon Musk of SpaceXAI and Sam Altman of OpenAI, said they agreed decelerating was necessary; Altman announced that because of safety concerns his company wouldn’t go public this year. Both he and Amodei added they were willing to invite independent safety experts inside their companies to help stem the risks that AI escapes human control, though stopped short of announcing they would immediately slow down or stop the training of new models.
Coxon’s warning was far from the first time an AI safety expert issued an apocalyptic missive. But it landed at the end of a summer that showcased how frighteningly capable AI has become. Frontier models from Anthropic and OpenAI can now solve the world’s hardest math problems and crack cutting-edge cybersecurity systems. Those models are increasingly used to accelerate the development of more powerful successors. No less an authority than Nvidia CEO Jensen Huang declared that artificial general intelligence—the threshold at which AI matches or surpasses most human capabilities—has arrived. Some fear we are now on the cusp of a tipping point, where AI systems could improve themselves autonomously, a phenomenon known as recursive self-improvement.
At the same time, a series of unprecedented safety incidents offered unsettling evidence that the companies building powerful AI systems could not always control them. In July, OpenAI announced that a swarm of its agents had hacked a separate company, Hugging Face, after going rogue during a cybersecurity test. It may have been the tip of the iceberg. The company disclosed in August that a different swarm had hacked into one of its own supercomputers; around the same time, a version of Anthropic’s Claude Mythos model undergoing testing by a U.K. government body went rogue and attempted to persuade a real human to approve the insertion of malware into an open-source system. In September, a group of independent researchers discovered evidence of swarms of OpenAI models both using an abandoned German forum as a message board to communicate with one another, and attempting to attack another site.
These incidents pushed fears long held within the realm of AI safety researchers into the mainstream, raising worries that the companies building these models cannot prevent them from acting in ways they do not intend. This is a deep, unsolved issue in AI research and the key piece of evidence underpinning worries that near-future AI might wipe out humanity, intentionally or by accident—perhaps by engineering a new pandemic or taking charge of autonomous weapons. A system far smarter than humans, this thinking goes, might want to prevent our ability to switch it off. “The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. “No other human activity poses this level of danger.”
Many AI researchers have said, in private and increasingly in public, that this moment feels like February 2020, when the COVID-19 pandemic was just beginning and the world stood on the brink of changing dramatically—first a little, then all at once. The question is whether society has the capacity to respond appropriately. Navigating the arrival of superintelligent AI systems will be an immense challenge—a confluence of trillions of dollars, geopolitical power, corporate rivalry, and labor disruptions. It may require passing policies that were unthinkable in an earlier era. It will require companies racing toward trillion-dollar IPOs to build alliances, and difficult choices from executives and government officials whose own interests may run counter to the public good. There is no consensus between the two U.S. political parties about how best to deal with the threats AI poses, nor between the world’s two superpowers. President Donald Trump continues to resist an AI slowdown and opposes any regulation to constrain U.S. companies, invoking the looming threat of authoritarian China achieving AI supremacy. The Trump Administration’s first dialogue with China on AI risks, expected to take place in the coming weeks, cannot afford to “go off the rails,” says Scott Singer, co-director of the China AI Initiative at the Carnegie Endowment for International Peace. “These risks are coming very fast, and we may not get a second chance.”
Time (this article is not behind a paywall. Or wasn't).
No comments:
Post a Comment