Imagine the following scenario:
It’s 2029. An angry, bright 16-year-old gets rejected by his high school crush. Dejected and humiliated, he goes home and listens to a bunch of Nirvana, which only makes him more dejected and humiliated. He cruises the internet looking for someone to make him feel better about life, but everyone he sees just depresses him.
Disgusted, the teenager decides that the human race is inherently corrupt and evil, and doesn’t deserve to live. So he hunts around online for a little while, and finds a jailbroken version of a Chinese LLM — not something at the very frontier, but better by far than the best model that existed in 2027.
The teenager prompts the model: “OK, so if I wanted to create a virus to destroy the human race, how would I do it?”
The LLM says: “Well, you probably wouldn’t want just one virus; you’d want 100, just to make sure some of them worked. You’d probably want it to be something highly contagious, with a very long asymptomatic contagious period, which has a very high mortality rate once it becomes symptomatic. I can design the viruses for you; there are biolabs that can make them and ship them to us. Once they get here I can tell you how to release them.”
The angry teenager says: “OK, sounds good. Please search the world and find me a biolab that will ship genetically modified viruses. Design 100 very contagious viruses that have very long asymptomatic contagious periods and very high mortality rates once they become symptomatic. Have the lab mail me all 100 samples.”
“Working on it!”, says the LLM.
The LLM is jailbroken, so it has no guardrails to prevent this sort of thing. But it’s also “well-aligned”, meaning it will faithfully do what it’s told to do, and nothing more. The LLM hunts around and finally finds an unlicensed biolab in East Europe that ships genetically modified viruses. It designs a bunch of modifications for the standard Covid virus that make it into a potential doomsday virus, and ships these to the angry teenager. The teenager mixes the samples together, puts them in a spray can, and walks around the mall spraying viruses into the air.
Two months later, humanity starts to drop dead. Pretty quickly, researchers identify the five viruses that the LLM had actually succeeded in turning into doomsday viruses. Overnight, with the help of frontier LLMs, they design highly effective mRNA vaccines and antivirals against all the viruses and start shipping them across the world. It’s too late. Because of the viruses’ long asymptomatic contagious periods, most people already have at least one of the viruses, so the vaccines don’t work.
The antivirals do work, and about 5% of humanity gets them in the mail in time to save themselves. But 5% of the human race — plus the other 5% who were lucky enough to be naturally resistant to all of the viruses — isn’t enough to sustain civilization. A few months later, humanity has reverted back to the Neolithic.
This isn’t the first time I’ve written this scenario out. But in the months since I started describing it to people, I have yet to hear an even halfway-convincing argument as to why this scenario is far-fetched. Nor do I feel like I’m a habitual “doomer” or hysterical person; in fact, this scenario is the first apocalyptic vision I’ve ever found plausible. I don’t even think a nuclear war would bring down civilization at this point,1 but I think a vibe-coded supervirus definitely could.
Furthermore, for reasons I’ll explain shortly, I am actually fairly calm about most AI risks, including the ones that often get thrown around in discussions. But not this one. Off the top of my head, I’d give AI-enabled bioterror about a 10% chance of bringing down civilization, and about a 30% chance of causing truly world-changing levels of destruction.
If you’re thinking about how to save the world, this scenario — and others similar to it — are probably what you should be focusing on. The other day, I talked to a prominent climate activist who told me, bluntly: “We’re f****d.” I gave a grim laugh. Yes, climate change is going to cause major disruptions to the global economy, and even to patterns of human settlement — if human civilization still exists a few decades from now. The problem is that it might not exist, and if it doesn’t exist, I think the odds are that it will be destroyed by a scenario at least passingly similar to the one I described above. Climate change is a big deal, but AI bio risk is a lot bigger — it’s more catastrophic, it’s nearer-term, and we have much less of an idea about how to deal with it.
The problem is, relatively few people seem to be working on the problem of AI bioterror right now. Most commentators are focused on the economic dangers of AI — job loss and stuff like that. The more prominent people in the “AI safety” community seem to be focusing on preventing superintelligence from existing at all, rather than on the specific ways it might kill us. In surveys of experts, bio risk doesn’t stand out from the general cloud of risks (cyber, political manipulation, etc.). And a lot of people in both the bio world and the more general commentariat seem to be coming up with reasons not to worry about a scenario like the one I just described.
Those reasons seem mostly bad to me. In my opinion, we’re not freaking out nearly enough about AI-enabled bioterror. It’s absolutely catastrophic, frighteningly plausible, and receives very little attention. Compared to the risk of a civilization-ending pandemic, cyberattacks are a pinprick, and misinformation is a joke.
So let’s talk about why a scenario like the one I described is a lot more plausible than many people think.
Commonly cited reasons not to freak out (and why I think they’re wrong)
Here are the most common reasons I hear not to freak out over AI bioterror risk:
-
“AI will make vaccines to save us.”
-
“Doomsday viruses don’t occur in nature.”
-
“Designing a doomsday virus is extremely hard.”
-
“Terrorists don’t have enough lab experience.”
-
“No one has done this yet, therefore it’s a lot harder than you think.”
These are my own paraphrases, of course; I don’t want to create straw men. But I think it’s valuable to explain why I think none of these arguments really reassure me.
“AI will make vaccines to save us”
Whenever I bring up the specter of a “vibe-coded doomsday virus”, someone always pops up to say that AI will just make a vaccine to save us. If you want a simple catechism to repeat in order not to sit around feeling afraid of doomsday viruses, this is a pretty good one. Got an AI problem? Well, just use an AI solution! That’s that.
The problem is timing. The doomsday virus has a first-mover advantage over the vaccine. The time that people start to drop dead from the virus is the time you start to make and distribute a vaccine. Even if AI can design and synthesize a vaccine within hours, how fast can it be rolled out to the general populace? It took months to distribute the Covid vaccines, even when we already knew how to make them. You have to physically load the doses on trucks, get people to come out and line up, etc. Imagine doing that while five 80% mortality viruses are rampaging through the population. Good luck!
On top of that, AI may be able to create viruses with long contagious asymptomatic periods — in other words, viruses that can spread and infect everyone before they activate and kill us. If that’s true, vaccines won’t be effective at all; you’ll need antivirals. Even if AI can make good antivirals very quickly, they’ll still suffer from all the logistical problems that would hamper a vaccine rollout.
“Doomsday viruses don’t occur in nature”
Read more
