What are self-replicating ‘AI worms’, and why is Anthropic worried about them?

The tech company says successful AI worms need more capability on less powerful devices, but current trends make that feasible by the end of the year

Warnings about what artificial intelligence might do in the future, together with alarming reports of what it has gotten up to already, are being issued so quickly that even security experts have difficulty triaging what to worry about.

Yet a report this week from Anthropic’s offensive research team grabbed attention by describing a new, practical threat, one devoid of human characteristics or apocalyptic musings. The red team, as such internal testers are called, said it created a malicious AI program that spread autonomously through internet-connected “smart” televisions and could do the same inside other ordinary devices. While machines running AI usually need a fair amount of computing power, the researchers said that their program split parts of itself among 15 televisions and was still able to function by using a bit of power from each.

In addition to evoking the horcruxes that kept pieces of Lord Voldemort’s soul alive within objects in the Harry Potter series, the report alarmed some professionals because the generally weak security of smart televisions and many other “internet of things” devices would be very hard to correct, even with a concerted effort.

“It’s something to worry about just like every other wave of cyber-attacks. We need to mitigate the possible effects,” said Joshua Saxe, who led development of Meta’s AI security effort. “I’m concerned right now that folks in the cybersecurity community are not taking these kinds of real risks as seriously as they should be.”

The damage of such “AI worms” could be astonishingly large. Self-replicating computer viruses, which need not trick a human to propagate, are known as worms, and earlier worms have wreaked some of the worst havoc in the history of the internet.

That legacy includes the Morris worm of 1988, which arrived before the mainstream consumer internet, when a graduate student released a program that took advantage of various software flaws and weak passwords, copying itself so rapidly that many infected machines slowed to a stop. The US and Israel brought worms into international espionage with the stealthy Stuxnet in the lat 2000s, which sabotaged Iranian nuclear centrifuges. Other worms like the Russian NotPetya and North Korean-attributed WannaCry have caused tens of billions of dollars’ worth of damage worldwide.

Many of the world’s most destructive worms have depended on known vulnerabilities that device owners have not patched, sometimes months after patches became available. The Anthropic team predicted that a worm like theirs could function in the same way and be harder to stop, because smart devices receive patches less often than phones or laptops. Worse, they theorized that an AI worm could keep adapting to other types of hardware by researching newly disclosed vulnerabilities.

What differentiates AI worms from typical reproducing viruses is that AI models can learn as they go, said Ariel Herbert-Voss, an early security expert at OpenAI who now runs AI security startup RunSybil. Existing cybersecurity infrastructure is not ready for the proliferation of such threats, Herbert-Voss said. “We need to better enable our security teams. We need to be giving them more resources to fix problems.”

Worms could also be used for benevolent purposes, suggested Chris Inglis, a former deputy director of the National Security Agency who served as National Cyber Director in the Biden White House.

“The question of whether AI worms are plausible has been settled by several breakout events over the past six months,” Inglis said by digital message. “The better question is how we will employ them, counter them, and/or combine them with other emerging technologies.”

Anthropic researchers said a successful AI worm would need to have more capability on less powerful devices than is possible right now, but they added that current trends will make that feasible by year’s end. Their research follows a paper published by experts at the University of Toronto a few months ago, which infected a full-size computer and could spread to similar machines more rapidly than the television worm.

“The capabilities have been developing very rapidly,” said co-author and Toronto PhD student Jonas Guan.

Guan and other experts said that, as a practical matter, an AI worm should not top anyone’s list of AI fears. The Anthropic version moves very slowly, Guan said, and worms have predictable and detectable actions that can be blocked at multiple layers. Basic security hygiene, including requiring strong passwords and two-factor authentication, would stop most worms and other hacking, experts said.

“Nothing about this is truly feasible or impactful,” said Juan Andres Guerrero-Saade, vice-president of research at security firm SentinelOne. “It’s nifty stunt hacking that’s getting way more amplification in service of the ridiculous doomer narratives that suggest models have a mind and will of their own.”

Outside Washington, many in the AI and security worlds believe greater government oversight of AI is now essential. But members of Congress have introduced mutually conflicting bills, while the White House has so far created only a small review board .

Security researchers and laypeople alike find themselves weighing the motivations behind fearful prognostications from AI companies, which range from espousing quasi-religious beliefs, to seeking legislation to exclude rivals or save money, to hyping the anticipated multi-trillion-dollar public valuations of the San Francisco AI giants.

“The claim that a lot of this is marketing is correct, but the reality dwarfs any PR,” said Gadi Evron, who is running a conference on AI security in San Francisco this month called Unprompted. “Whatever claims the labs are making, they are not warning us enough. We need to understand that nobody can stop the AI storm.”