Many years ago, I too worked in AI. Compared with today’s Large Language Models (LLMs) — such as ChatGPT, Gemini, and others — our work was embarrassingly primitive. We were building Expert Systems to help with the diagnosis of software issues. They were called expert systems because we hoped to tease out of human experts their problem-solving rules that we could then encode and apply. The challenge expert systems faced, in fact the challenge all AI systems face, was how to capture, encode, and apply knowledge.

We failed. Miserably. Or spectacularly, depending on how you look at it.

Where chatbots (LLMs) have succeeded

This compound challenge — capturing, encoding, and applying knowledge — is the specific problem LLMs (chatbots) have solved in many cases. Rather than relying on humans to cough up what they know, LLMs break the problem into two phases. First, training captures the semantics of words such that each word mathematically relates to other words sharing similar concepts. For example, the encoding for King relates to Male as Queen relates to Female. The relationship is mathematical in that you can, literally, subtract Male from King and add Female to get Queen. This technique — called Word Vectors — is brilliant and well-suited to computer manipulation.

The second problem LLMs have largely solved is knowledge acquisition; the capturing of all those “rules” we tried to pull from experts. Asking humans fails because we often do not clearly know what we know until we need to use it.

LLMs don’t ask, they scan. The systems “look at” human output — literature, legal documents, medical diagnoses, and more — over and over, and over again, slowly building up relationships between terms.

LLMs are pattern catchers. They catch patterns, patterns of patterns, and patterns of patterns of patterns. Some are simple patterns: subject-object-verb is a simple pattern. For example, “sunlight melts ice cream.” Many of these top-level or obvious patterns build upon other patterns. For example, our subject-verb-object pattern relies in part on punctuation. Which is why

John hit Steve.

means something different from

John hit! Steve!

All our communication is a collection of patterns at different levels. Punctuation relies on the deeper rules of grammar, speech, ideas, and modes of expression. Even abstract concepts in math, philosophy, law, and similar areas are patterns. If an LLM “sees” enough of a pattern, it encodes it as useful. And an LLM will “see” patterns we miss (or ignore or take for granted) because it “looks at” everything thrown its way.

Sometimes it “sees” patterns that are not there. For example, it may see a correlation between breathing and dying of cancer (since everyone who dies of cancer breathes air). But it may also “see” real patterns invisible to us, beneath the surface. Any correlation it encounters enough will, with some level of strength, get saved for later use.

Understanding how LLMs work goes a long way toward explaining why LLMs fascinate and surprise us. They surprise us because, well, we’re kind of lazy. We take many of the patterns in our lives for granted. We rely and act on them without thought. Some we may be wholly unaware of. It takes us by surprise when an LLM returns one these in a response only because we weren’t looking closely.

It is also why LLMs sound so human. LLMs store all the patterns by which humans communicate. And without patterns — in our language, the way we use language to express thoughts, to convey emotion — we could not communicate. LLMs, far more than my measly attempts with Expert Systems, excel at capturing and applying these patterns.

What chatbots can’t do

No matter how clever they seem, LLMs are regurgitation machines. To spice up the output, developers mix a bit of randomness into how LLMs generate a response; otherwise, they’d likely spew the same response to the same query each time. This randomness can fool us into thinking an LLM acts creatively.

But LLMs are not creative in the sense of “seeing” new and interesting connections. They emit new connections by accident, by randomly altering the relationships “learned” during training. These unexpected results can feel creative because we had not previously seen them.

Surprise and creativity, however, are not the same. Surprise comes when something is unexpected. Creative output is unexpected and, so, surprising. But not all surprising things grow from creativity. Rolling a pair of sixes five times in a row is surprising, but it’s not creative. An LLM, through randomness or due our limited knowledge, generating a connection new to us is not acting creatively. It is throwing stuff at the wall and seeing what we let stick.

Since LLMs capture and encode patterns through repetition — that is, saving what they “see” most often — they are, mathematically speaking, holding onto the average, that which occurs most often in a context. Research bears this out. The work of poor performers improves much more than that of good performers. The writing of poor writers gets better when assisted by an LLM more than the writing of good writers does.

This is what we should expect: Poor performers are, by definition, below average. An LLM helps them raise their work quality to average or slightly above. That same research shows that the work of good performers, if they’re disengaged with how they use the LLM, degrades over time, dragging them downward. Further, other research suggests that the output of LLMs degrade as they consume increasing quantities of LLM generated output.

And here lies the danger: Unless we continue to value human-created content, even if that creation amounts to editing the output from an LLM, we’ll find ourselves locked in a spiraling descent into a mediocre word-salad. It will soften our thinking. It could dull our creativity. Machines and humans will meet in a warm pablum of blah. Despite the unfounded predictions of Terminator-style AIs, the usefulness of LLMs or the damage they cause depends, as with every other technology, not on the technology but on how we use it.

Human creativity is irreplaceable. LLMs can help spur creativity; they cannot replace it. Hammers do not build a house; we do. We must use our tools with our eyes wide open, or else we’ll become like them.

You may also wish to read: Model collapse: AI chatbots are eating their own tails. Meanwhile, organizations that laid off writers and editors to save money are finding that they can’t just program creativity or common sense into machines.
The problem is fundamental to how they operate. Without new human input, their output starts to decay