In 2016, tens of thousands of computer-security researchers gathered at the Bally’s Las Vegas for DEF CON, the world’s largest hacker convention. During the four-day conference, attendees witnessed talks and demonstrations on subjects such as how to remotely control an airliner, disarm electronic safe locks, or bypass security on an ATM. But that summer the conference’s organizers wanted to try something sensational, even by the standards of an event where hacking electronic polling stations is considered normal. Instead of humans hacking computers, they wanted computers to hack each other. What ensued foreshadowed a future of vulnerability to computer-based hacking, and violation of information systems on a scale previously unimagined.
In partnership with the Defense Advanced Research Projects Agency (DARPA), a branch of the U.S. Department of Defense that funds research on breakthrough technologies, DEF CON hosted this first—and to date only—Cyber Grand Challenge, a hacking competition where artificial-intelligence systems competed to autonomously hack computer programs defended by other AIs. The competition was structured as a game: the AI that was best able to find and exploit vulnerabilities in other systems while protecting its own system from being hacked during the 10-hour competition would earn its creators a $2-million prize.
From a human’s perspective, the AI-only hacking competition didn’t look like much. A half-dozen brightly colored server racks running sophisticated AI systems were arranged in a semi-circle on a stage in one of the hotel’s ballrooms; flickering LED lights on each machine were the only indicators that an all-out robot war was raging on DARPA’s network. But for Bruce Schneier, a computer-security expert and adjunct lecturer in public policy at the Harvard Kennedy School, what transpired that day was a sobering glimpse of a not-too-distant future when AIs can find and exploit vulnerabilities with superhuman speed, scope, scale, and sophistication. These future AI hackers won’t be limited to computers. They will hack financial, political, and social systems in unimaginable ways—and people might not even notice until it’s too late.
As Schneier details in his new book, A Hacker’s Mind (W.W. Norton), humans began gainfully exploiting vulnerabilities in systems—hacking—long before there were any computers. Any time there is a complex process constrained by rules and norms, people will find ways to profitably subvert the goals of that system without breaking it. Whether those systems are computers, frequent-flier programs, professional sports contests, or democratic institutions doesn’t matter. All have vulnerabilities.
It’s hardly surprising that humans would turn to artificial intelligence to develop better hacks. AIs never rest and they can analyze massive amounts of data to find patterns or information that might escape a human’s notice. These would be valuable skills for any hacker, but Schneier emphasizes an even more important distinction: AIs don’t think like people do.
At the most basic level, an AI is just a sophisticated software program that uses step-by-step procedures—algorithms—to solve a narrowly-defined problem, such as identifying an object in an image. What makes AIs different from other computer programs is that the more data they process, the more adroit they become. Like humans, AIs are designed to learn from their experience. Unlike most humans, however, AIs aren’t constrained by norms, values, and assumptions. This means they can devise novel hacks that humans would never consider. Schneier offers an example of a computer programmer who wanted his robot vacuum to stop bumping into objects as it cleaned his house. He trained the AI program running on the robot to avoid triggering the bumper sensors on the front of the vacuum, but instead of learning to avoid objects, the AI learned how to drive backward. This hack meant the AI technically accomplished what the programmer designed it to do—avoid actuating the bumper sensors—but in such a way that it didn’t fulfill his goal of avoiding objects. “The thing with AI hackers is that it’s not just that they’re faster or that there’s more of them,” he says. “They’re a different animal.”
Schneier sees two primary ways that AI hackers pose a threat to financial, political, and social systems. First, an AI may be explicitly instructed by its designer to find and exploit vulnerabilities in a system. For example, the developer might feed the AI all the world’s tax codes and instruct it to find the most profitable loopholes. But a second threat is of greater concern, he says. Like the programmer’s robot vacuum, an AI may inadvertently hack a system by finding a “solution” that its designers never intended. This kind of unintentional hack is especially troubling because it might occur and remain undetected.
For now, these types of AI hackers are science fiction, but as Schneier puts it, “It’s not stupid science fiction.” The winner of the DARPA Cyber Grand Challenge, an AI called Mayhem, got trounced when it entered an all-human version of the same hacking competition. But during the past few years it has rapidly improved and is now used by the Defense Department. AI hackers don’t require any breakthrough technologies; the key pieces already exist and just need someone to put them together. Schneier is reluctant to predict when the first fully autonomous AI hackers will begin operating, but he says that will probably occur sooner than anyone thinks. He points to the example of Go, a game that most experts thought AI could never master—until DeepMind’s AI beat the world’s best player for the first time in 2016. “The thing about AI is that it’s discontinuous,” he says. “It doesn’t progress linearly in ways you can nicely predict.”
He is certain that human institutions aren’t yet equipped to handle an onslaught of AI-devised attacks. Software companies can rapidly deploy new code to fix a vulnerability after it’s discovered, but human systems change much more slowly. If we’re going to survive the age of AI hackers, we’ll need processes that can plug social, economic, and political loopholes as fast as AIs identify and exploit them. Schneier isn’t sure how these new systems governing our lives will operate, but he says effective solutions will have to be fast, inclusive, transparent, and agile.
Ironically, AI hackers themselves may offer a first line of defense. Although people tend to think of hacking as unethical or even criminal, hacks can also be a critical engine for progress. If a hack benefits the users of a system, that system’s administrators may formally adopt and normalize the hack, thus neutralizing it. As Schneier points out in his book, many aspects of everyday life that we take for granted—such as dunking in basketball, and banking certificates of deposit—began as hacks. Whether a hack is harmful or beneficial is often a matter of perspective, but every hack is critical to the evolution of a system. AI hacking is no different. Schneier says he can envision a future where AIs can make software, regulations, and other systems more hack-resistant by searching for vulnerabilities before they are deployed in the real world.
“At its core, hacking is a balancing act,” he concludes in A Hacker’s Mind. “Unless we can hack the process of hacking itself, keeping its benefits and mitigating its costs and inequities, we may struggle to survive this technological future.”
