Two philosophers, Joe Slater and Michael Townsen Hicks, and political scientist James Humphries, wrote an article in Scientific American earlier this moth with the arresting title, “ChatGPT Isn’t ‘Hallucinating’—It’s Bullshitting!, in which they talk about the increasingly familiar problem of hallucinations — Soviet bears in space and all that.

They take issue with the term hallucination:

It has become common to describe these errors as “hallucinations.” But talking about ChatGPT this way is misleading and potentially damaging. Instead call it bullshit. We don’t say this lightly. Among philosophers, “bullshit” has a specialist meaning, one popularized by the late American philosopher Harry Frankfurt. When someone bullshits, they’re not telling the truth, but they’re also not really lying. What characterizes the bullshitter, Frankfurt said, is that they just don’t care whether what they say is true. ChatGPT and its peers cannot care, and they are instead, in a technical sense, bullshit machines.

Joe Slater, James Humphries, Michael Townsen Hicks, “ChatGPT Isn’t ‘Hallucinating’—It’s Bullshitting!,” Scientific American, July 17, 2024

They explain how the models work (a sophisticated form of text prediction, as in cell phones) and note,

Now, we can see from this description that nothing about the modeling ensures that the outputs accurately depict anything in the world. There is not much reason to think that the outputs are connected to any sort of internal representation at all. A well-trained chatbot will produce humanlike text, but nothing about the process checks that the text is true, which is why we strongly doubt an LLM really understands what it says.

Slater, Humphries, Hicks, “It’s Bullshitting!”

That’s why it’s worrisome that so many people rely on such machines for writing shortcuts. It is somewhat like relying on the town gossip for information. The tale might be true but…

Slater, Humphries & Hicks have also authored an open-access paper on this topic, similarly titled “ChatGPT is Bullshit” In which they warn against the tendency to assume that anything the chatbot does is based on understanding (it’s not):

Calling chatbot inaccuracies ‘hallucinations’ feeds in to overblown hype about their abilities among technology cheerleaders, and could lead to unnecessary consternation among the general public. It also suggests solutions to the inaccuracy problems which might not work, and could lead to misguided efforts at AI alignment amongst specialists. It can also lead to the wrong attitude towards the machine when it gets things right: the inaccuracies show that it is bullshitting, even when it’s right. Calling these inaccuracies ‘bullshit’ rather than ‘hallucinations’ isn’t just more accurate (as we’ve argued); it’s good science and technology communication in an area that sorely needs it.

“ChatGPT is Bullshit” Hicks, Michael & Humphries, James & Slater, Joe. (2024). ChatGPT is bullshit. Ethics and Information Technology. 26. 1-10. 10.1007/s10676-024-09775-5.

One way of looking at it: To hallucinate – to see things that aren’t there – you would need the ability to see things that are there. But the chatbot is not seeing or not seeing anything. It is regurgitating nonsense just as it regurgitates sense.

Also an item in Nature last week tackled the problem of model collapse, as discussed in a recent open access paper:

Training artificial intelligence (AI) models on AI-generated text quickly leads to the models churning out nonsense, a study has found. This cannibalistic phenomenon, termed model collapse, could halt the improvement of large language models (LLMs) as they run out of human-derived training data and as increasing amounts of AI-generated text pervade the Internet.

“The message is, we have to be very careful about what ends up in our training data,” says co-author Zakhar Shumaylov, an AI researcher at the University of Cambridge, UK. Otherwise, “things will always, provably, go wrong”. he says.” The team used a mathematical analysis to show that the problem of model collapse is likely to be universal, affecting all sizes of language model that use uncurated data, as well as simple image generators and other types of AI.

Elizabeth Gibney, “AI models fed AI-generated data quickly spew nonsense,” Nature, July 24, 2024.

Readers may have heard of this as the jackrabbits problem: In one test situation, the last iteration of the chatbot asked about architecture was, for unknown reasons, repetitive nonsense about jackrabbit tails.

What they’re saying is that as more AI-generated gunk populates the internet, the chatbots will recirculate it into utter nonsense. From the paper:

Our evaluation suggests a ‘first mover advantage’ when it comes to training models such as LLMs. In our work, we demonstrate that training on samples from another generative model can induce a distribution shift, which—over time—causes model collapse. This in turn causes the model to mis-perceive the underlying learning task. To sustain learning over a long period of time, we need to make sure that access to the original data source is preserved and that further data not generated by LLMs remain available over time. The need to distinguish data generated by LLMs from other data raises questions about the provenance of content that is crawled from the Internet: it is unclear how content generated by LLMs can be tracked at scale.

Shumailov, I., Shumaylov, Z., Zhao, Y. et al. AI models collapse when trained on recursively generated data. Nature 631, 755–759 (2024). https://doi.org/10.1038/s41586-024-07566-y

It’s a good thing that the public is learning about the genuine limitations of chatbots and other forms of artificial intelligence. It might, of course, take a disaster to drive the reality home.