Eliezer Yudkowsky and Nate Soares, the authors of  If Anyone Builds it, Everyone Dies: Why Superhuman AI Would Kill Us All (2025), assert that AIs will develop independent minds, with their own goals and desires (if they haven’t already). And, when they do, it’ll be bad for us. I’ve been looking at their claim in terms of two things: its origin in the Doomer perspective on AI in Silicon Valley and its relationship to what known AI actually does. Now I want to turn to the question of what it means to have a mind.

 What is a mind? Can just anything have one?

First, I should say that I am not a philosopher of mind. But in fairness, neither is Yudkowsky or Soares.  Common sense would suggest however that self-awareness seems a minimum requirement for having a mind. Desires and goals require a sense of self. There must be something that it is like to be you, having that desire, as the philosopher Thomas Nagel puts it.

Yesterday, I pointed out that modern AI models cannot hold meaning. For text generative models, such as ChatGPT, this follows from the fact that meaningless tokens form the basis of their training data. The same situation holds for similar models designed to generate images.

Just for now, however, I’ll grant the authors the fantasy that these models do capture meaning rather than just correlations. I’ll pretend that, somehow, they truly hold in their networks representations of meaning. Why grant this? Because, as you will see, it actually doesn’t matter.

Also, since Yudkowsky and Soares are materialists, I’ll restrict myself to the dominant materialist views on minds and consciousness. Again, why this concession? Because even though they believe that machines can be made to think, I doubt that they believe they get endowed with souls. So, again, it won’t matter.

How materialists think the mind emerged

There are two dominant materialist models of mind: emergence and panpsychism.

Emergence — Nothing to see here

Emergence — generally known as epiphenomenalism — considers the mind a by-product, the afterglow, of whatever is going on in the brain. We are not truly conscious, we just think we are. And we do not have goals, desires, or plans. We apply these labels, after the fact, on actions caused by the chemistry of our brains. The action was going to take place, label or no label.

If this is, in some sense, an accurate description of what takes place — then you could hope to replicate the effect in an artificial brain, such as an AI. And, if that mind arises from “knowledge” — those patterns of facts that our brains collect about the world — then perhaps a sufficiently well-trained AI would, as a natural result, develop a mind.

If Yudkowsky and Soares take this approach, it’s a dead end. It is a statement of faith, not fact. Even if such a machine mind succeeded, it would fall into the cause-and-effect conundrum: the unbroken chain of events from the Big Bang until now. Goals, desires, and plans do not exist; only post-event labeling of predetermined events.

Let’s take this further and assume that our minds do arise from our brains. If so, then the arrangement of stuff in our brains matters. But what stuff? The organic material or the patterns it holds? Since we have consciousness before we learn, it cannot be the patterns (that is, it cannot be what we learn). It must be the arrangement of organic material — the neurons, synapses, chemical messengers, and so forth. It is a long leap of faith, then, to believe that a machine, which lacks all those, could have a mind.

It is a hopeless fantasy to believe a mind will magically emerge in a machine filled with patterns extracted from mountains of data. But, then, it’s hopeless for organic brains as well, which is why the other view I referred to above, panpsychism, is gaining prominence.

Panpsychism —consciousness is everywhere

Panpsychists address the shortcomings of emergence theory by holding that consciousness (whatever that is) is a property of the material world, just like atoms and gravity. We can think of it as a “woo” answer to the shortcomings of emergence:

I’ve come across two forms of this position: One suggests that, if you pile enough stuff together (no one is clear on just how much), enough “consciousness” will be present to form a mind. David Chalmers seems to adhere to this view. He once toyed with the idea that even a thermostat could be conscious.

This unlikely view assumes that mind derives from the arrangement of matter. To assume that the arrangement of electrons “holding” the information within a modern AI is the right arrangement to form a mind is a leap in the dark. Philosophers can muse over whether my smart thermostat is a lot smarter than it appears. The only minds for which we have reasonable evidence all entail having a brain, which is a pretty specific arrangement of stuff.

The other panpsychist view that I’ve encountered is the view that our brains act as a receiver of consciousness. It’s not so much that we have consciousness as we are tuned into the consciousness present throughout the universe.

There is no evidence that an AI could act as such a receiver. It’s another long leap from understanding the brain as a receiver to believing that a collection of computers, without physical correspondence to brains, could do the same thing.

There’s actually nobody home

The belief — and fear — Yudkowsky and Soares market, that an AI has or can develop a mind is wishful thinking (at best). But why is their view so seductive? Why do so many get convinced so easily?

In 1966, MIT computer scientist Joseph Weizenbaum released ELIZA: an early experiment in natural language computation. ELIZA relied on pattern matching and substitution to emulate conversation and create the illusion of understanding.

The most famous script ELIZA employed simulated a Rogerian psychotherapist. This was a clever choice. The Rogerian school of psychotherapy avoids directly challenging a client, believing that the answers the client sought lay within themselves. The psychotherapist’s role, then, is to tease these answers out of the client through unconditional positive support. ELIZA emulated this approach through a question‒answer exchange triggered by patterns.

Although ELIZA was not a psychotherapist, Weizenbaum’s own secretary reportedly asked him to leave her alone with the program so she could have “a real conversation” with it. Weizenbaum later wrote:

I had not realized … that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.

Our tendency to see minds where they do not exist — as Weizenbaum’s secretary did with ELIZA — tells us more about ourselves than about the machines that can fool us. The only minds we encounter reside in other living beings. We often infer these minds from conversation alone (for example, a text or phone conversation). When a machine successfully imitates that kind of conversation — which ChatGPT was designed to do — it almost instinctively triggers us to “see” another mind where none exists.

The fear of a modern AI developing a mind that will lead to goals, desires, and plans bad for humanity is a pipe dream (nightmare?) with no basis in the real world. Some people, however, need repeated reminders: There’s no one home; it’s just a machine.

If the author’s fears of lethal AI are unjustified, are we free and clear, meaning that AI poses no existential risk?

On this point, I am much more in agreement with Yudkowsky and Soares. Tomorrow, I’ll argue that modern AIs do pose a serious risk — just not for the reasons they give.

Next: Yudkowsky and Soares are not grappling with the real threat AI poses to us: widespread abuse

Here are the first two parts of my look at the arguments in this thought-provoking new book:

Fearing the Terminator, Missing the Obvious. In Part 1 of my review of the new AI Doom book, If Anyone Builds It, Everyone Dies, we look at how the authors first developed the underlying idea. By 2020, authors Yudlowsky and Soares were already Doomers but the rapid success of ChatGPT and similar models heightened their worries.

and

Fearing the Terminator: Does current tech warrant the doomsaying? People will worry less if they understand why the text generation programs not only do not think but in fact cannot think. Bottom line: Text generative AIs do not capture meaning. They capture relationships which approximate meaning well enough to be useful.