In 1980 philosopher John Searle proposed a thought experiment, now known as the Chinese Room argument: Suppose that a computer in a closed room receives questions in Chinese characters and generates output in Chinese characters, doing such a good job that a native Chinese speaker is convinced that the computer is human. Now suppose a human who does not read or write Chinese takes the place of the computer. The human has access to the computer’s code and can follow the instructions the computer uses to generate answers written in Chinese to questions written in Chinese. So the human gives exactly the same answers that the computer gives. Does this mean that the human understands Chinese in any meaningful way? If the answer is no, how can we say that the computer understands Chinese?

Sound familiar?

ChatGPT and other large language models (LLMs) generate human-like responses to prompts even though they do not understand what any of the words in the prompts mean. Yet, enthusiasts proclaim that LLMs have achieved or soon will achieve artificial general intelligence (AGI)—the ability to perform any intellectual task that humans can.

In October 2023, Blaise Agüera y Arcas and Peter Norvig wrote a piece titled “Artificial General Intelligence Is Already Here.” In an April 2024 podcast interview, Elon Musk asserted that, “My guess is that we’ll have AI smarter than any one human around the end of next year.” On November 11, 2024, OpenAI’s Sam Altman similarly predicted the arrival of AGI in 2025.

It is now clear that pre-training (a statistical analysis of text datasets) will not give us AGI. Not understanding what words mean, LLMs that rely solely on statistical patterns are prone to generating factual mistakes and nonsense; for example, claiming that the Russians have sent bears into space and that the game of tic-tac-toe is changed by rotating the grid 90 degrees to the right or left.

Recognizing this problem, thousands of people are now tasked with post-training LLMs in order to patch up known mistakes. This post-training improves the accuracy and usefulness of LLMs but does not make them intelligent in any meaningful sense of the word.

LLMs and the Monty Hall problem

I recently experienced a compelling example. The well-known Monty Hall problem is based on a game played on the television show Let’s Make a Deal (1963‒2003), hosted by Monty Hall (1921‒2017). A contestant is asked to choose one of three doors — there is a grand prize behind one door and goats behind the other two doors. After the contestant picks a door, Monty Hall always showed a goat behind a door that was not chosen and asked if the contestant wanted to switch doors.

Many people think that, because there are two remaining doors, the chances are now 50–50. However, we already know that at least one of the doors that wasn’t selected has a goat behind it. Does it matter if Monty Hall reminds us of this fact, or proves it by showing us a goat? We haven’t learned anything useful, so there is still a one-third chance that the selected door is the winning door. Therefore, the probability that the other closed door is the winner has risen to two-thirds. The contestant should switch.

ChatGPT and other LLMs have trained on numerous retellings of the Monty Hall problem and reliably generate the correct one-third answer. However, if the number of doors is changed from three to four, the probabilities change. The probability that the grand prize is behind the selected door is now one-fourth, leaving a three-fourths probability that the prize is behind one of the other doors, with each having a three-eighths probability of being the winning door.

For a while, ChatGPT and other LLMs failed when the number of doors changed. They either gave the answer to the original 3-door problem or gave nonsensical answers. Last week, I gave the 4-door problem to ChatGPT, using GPT-4o mini, and it gave the correct answer. It also answered correctly when I changed the number of doors to 5 or 10. In each case, it followed a chain of calculations that it had evidently incorporated during post-training:

               With n doors, the probability that the prize is behind the selected door is 1/n.

               This probability stays at 1/n when another door is opened.

               The remaining probability is divided equally among the other remaining doors.

For example, the probability stays at one-fourth for four doors, one-fifth for five doors, and one-tenth for ten doors.

Impressive. However, ChatGPT still does not understand the problem, as demonstrating by changing the number of doors to two.

Gary: There are two doors. One randomly chosen door has an expensive car behind it; the other door has a goat. You choose door 1. The host shows you that there is a goat behind door 2 and asks whether you want to change your door selection. What is the probability of winning if you switch?

If there is a goat behind door 2 then the prize must be behind door 1. Not understanding the words in the prompt or how they relate to the real world, ChatGPT instead followed the chain of calculations it had been trained to use and gave a nonsensical answer:

ChatGPT: This is a variation of the Monty Hall problem, but with only two doors instead of three. Let’s analyze the probabilities.

Initial Probability: Since one of the two doors was randomly assigned the car, the probability of initially picking the car is 1/2, and the probability of picking the goat is also 1/2.

After the Host’s Action: The host always reveals a goat. Since there are only two doors, once the host reveals a goat behind Door 2, the remaining door (Door 1) must either have the car or a goat (which we already know it does not).

Effect of Switching:

● If your initial choice was the car (1/2 probability), switching makes you lose.

● If your initial choice was the goat (1/2 probability), switching makes you win.

Thus, switching and staying have an equal probability of winning: 1/2. Unlike the classic Monty Hall problem (which has three doors), here switching does not improve your chances — it remains a 50-50 chance either way.

As in the Chinese Room thought experiment, following instructions can create an illusion of intelligence. However, an ability to follow instructions is a very limited form of intelligence and far from what is generally meant by AGI. When we ask a pocket calculator to tell us the square root of 49, do we count a correct response as evidence of intelligence?

ChatGPT will no doubt soon be trained to answer the two-door question correctly. With ongoing post-training to patch up glitches, LLMs are unquestionably becoming more reliable and useful for many tasks. But they are no closer to true intelligence — which includes the ability to make decisions when there is no instruction manual. The post-training cannot possibly anticipate all the situations in which an LLM might be asked to provide recommendations or make decisions. Nor can it anticipate the relevant information needed at that future date to consider the possible outcomes of decisions and assess the uncertainty about those outcomes; for example, whether to accept a pre-trial settlement offer or to go to trial.

No amount of pre-training or post-training will alter the fundamental fact that the real danger today is not that computers are smarter than us. Rather, it is that we think computers are smarter than us and consequently trust them to make decisions they should not be trusted to make.