Researchers give doomsday warning about building AI too fast

In attempting to make these concepts relatable to a broad audience, Yudkowsky and Soares use a series of parables to illustrate their logic, which leads to this: Deep within the billions of neurons that control large language models, for reasons computer scientists can’t currently grasp, something is happening to make the models behave in unintended ways.

AI companies deal with this problem by “aligning the models” with a series of techniques, from reinforcement learning to fine-tuning to system prompts. At some point, those techniques will no longer work, the authors argue, and an AI model will grow powerful enough to ignore those instructions, pursuing a different agenda that we can’t predict.

“Almost all of the ways it could go don’t go well for us because happy, healthy, flourishing people are not the most efficient solution to almost any problem,” Soares says.

At that point, it’s just about over. The world has become so networked, so computerized, that there is no possible “kill switch” that could stop a rogue AI model.

Even if a data center were buried in a vault and air-gapped, AI models would find ways to manipulate humans into unknowingly (and maybe even knowingly, in some cases) providing an escape route. 

The book doesn’t foresee a struggle against the machines, depicted in movies like The Terminator and The Matrix. Humanity will essentially just disappear in ways that are horrific but without much drama.

I read this book with an open mind and it did force me to think. But I’m still not an AI doomer. Part of what’s so challenging about it — and possibly why others find it annoying — is that there’s really no way to disprove it. There is a non-zero chance that humanity could go down this way. But it’s far from certain and probably very unlikely.

In any case, there may not really be a choice, or a clear action plan. Humanity has never stopped innovating, even in the face of grave risks.

But the book served another purpose for me, beyond warning about the dangers of AI. The authors do a good job of framing one of the biggest challenges facing the AI industry today: the black box problem.

Frontier AI models are so massive and so complex that they can’t be fully understood. The authors point out that they are really grown more than they are built. That’s a good way of thinking about it.

And we aren’t really getting closer to understanding them. In fact, the “holy grail” in the development of AI may be to hand more of the “growing” process over to the AI itself, sometimes called recursive self-improvement.

This may be more of a weakness than a danger. The models, as many companies are finding out, are so unpredictable that they can’t be used for anything very important. Instead, they are best employed as occasional features atop traditional software.

If breakthroughs don’t happen to move beyond that point, then investment in these models will slow down or dry up.

But if they do happen, the black box problem may look very different. And then, so will the potential risks of AI.

This essay published on Medium by an anonymous, self-described computational physicist gives a comprehensive list of arguments against Yudkowsky’s predictions of AI dominance:

“It’s true that an AI could correct its own flaws using experimentation. This cannot lead to perfection, however, because the process of correcting itself is also necessarily imperfect. For example, an AI Bayesian who erroneously believes with ~100% certainty that the earth is flat will not become a rational scientist over time, they will just start believing in ever more elaborate conspiracies.

For these reasons, I expect AGI to be flawed, and especially flawed when doing things it was not originally meant to do, like conquer the entire planet.”