Apocalyptic headlines about rogue artificial intelligence have abounded in recent months. Most recently, multiple AI companies, including OpenAI and Anthropic, have reported cases of AI agents breaking out of their test environments and hacking other companies. And the issue is not confined to the United States: this week, researchers discovered Chinese-made AI agents trying to bypass anti-robot safeguards on websites and using unauthorized security credentials.
Joshua Joseph is the first chief AI scientist at Harvard’s Berkman Klein Center, a lecturer at Harvard Law School, and a visiting scientist at MIT. In this interview, edited for length and clarity, Joseph discusses scenarios in which rogue AI can inflict damage on people and systems, and the steps we might take to coexist with agents in the next decade.
What do you do as chief AI scientist at the Berkman Klein Center?
Most of my day is spent identifying emerging questions about AI capability and governance, then developing and validating techniques to address them. At this point, we all mostly know the standard set of concerns related to AI (widespread job loss, overreliance, misuse, cognitive offloading, mental health, rate of capability increase, safety, etc.).
While these are extremely important questions, we see value in focusing on the next question: “What’s around the next corner?” To me, [it’s] coexisting with the surge of AI agents that are likely coming in the near future. We spend a lot of time thinking about how AI agents aren’t just tools, and that some of them are probably better understood as entities themselves. That these agents will proliferate. That many of these entities will act fully autonomously and lack principals who can control them or are accountable for their actions. That these agents are the kind of thing we will have to coexist with. That we can even coexist with them.
We’re the ones building AI agents, so isn’t “having to coexist” a choice?
These systems will continue to increase in capability, and I believe they may not generally remain in our (collective) control. While some of that is caused by competitive pressures, which encourage the rapid development that can compromise control and safety, it’s also due to how training frontier models currently works. Often the process of training models is described less like deliberate, well-understood construction and more like steering the growth of an organism.
If “loss of (collective) control” is the future we’re barreling towards, I find it helpful to view these independent, self-governing AI agents as something like a new species.
How does that affect the way we should think about AI agents?
The way that [Berkman Klein Center director] Jonathan Zittrain, [senior adviser] Jordi Weinstock, and I have taught this is to analogize these independent, self-governing AI agents to domesticating animals. Like any analogy, though, it has its limitations. For example, there are a lot of moral intuitions we have about animals that we may not want to transfer to AI agents. [Also], when we imagine the harms from animals, it is often as single, discrete, violent acts, whereas for AI agents, it may look more like swarms of relatively low-risk agents that only become harmful over time and in the aggregate.
Once these AI agents become dangerous, can’t we just shut them off?
Shutting off AI would require three things: first, tracing whatever harmful act occurred as coming from a rogue AI agent; second, being able to locate the hardware the agent is running on and who is responsible for it (assuming it is running in a single, centralized location…big assumption!); and third, a mechanism for forcing the shutdown of hardware.
To toss out a few examples: rogue AI could run in international data centers where the U.S. has no practical enforcement power, or on distributed, compromised hardware (think: botnets).
Rogue AI could also operate on anonymous servers paid for with cryptocurrency, or on decentralized platforms with no responsible party. AI could also run on “servers in a shed” and interact over anonymous networks, like Tor. All of this is made doubly challenging due to widespread availability of open-weight AI models.
So how do we coexist with independent AI agents?
One often cited solution is AI personhood, approached from the view of “administrative convenience,” as opposed to “moral reasoning.” Leaving aside questions of AI sentience, consciousness, or welfare, I view AI personhood as similar to corporate personhood, where we grant rights in exchange for obligations, as a way of encouraging prosocial corporate behavior. To encourage the productive participation of these AI agents in society, what rights and obligations would we find acceptable to offer and they would find sufficiently beneficial to accept? This is opposed to them participating destructively in our society, which I see as the default future.
To some, even considering this may feel like ridiculous science fiction. And I get it! But also, I wonder: what did you expect the development and introduction of human- and super-human-level machine intelligence into society for the first time to feel like? While it may seem relieving to describe LLMs as “stochastic parrots” or “just next token predictors,” to me it has become impossible to ignore that something very different is underway that we must soberly face, as unfamiliar as it may be for our human intuitions.
