In the 2000s, I was fascinated by the question of how to imbue search engines with some appreciation of serendipity. This is a bit like squaring the circle, as it turns out. But the general ambition was clear enough: sometimes, when searching for something — a pair of keys, a word on the tip of our tongue, who was president in 1960, the chemical formula for iodine — we end up finding something else.

The “something else” is a surprise because we weren’t looking for it, or at least didn’t think we were. And yet, it turns out to be exactly what we wanted — or more compelling than what we originally sought.

We’ve all had the serendipity experience, even online — clicking through a chain of links, scanning Google search results, drifting between loosely connected ideas. But search engines and information retrieval systems aren’t designed to enhance serendipity. They are designed for accuracy — for retrieving exactly what is implied by the keywords. In other words, they return what we want. What we are looking for.

If only we always knew what that was.

Image Credit: andreybiling - Adobe Stock

Serendipity is a major force in science, discovery, and the open-ended nature of thought itself. The famous cases remain compelling: Fleming didn’t set out to discover penicillin, Kekulé’s benzene structure came to him in a dream, and Gödel, lingering in the Vienna Circle, wasn’t supposed to uncover the limits of formalism but did. These moments fascinate not only because of what was found, but because they reveal how discovery actually works — not always through direct search, but through unexpected encounter.

Why serendipity fell into disuse

By the mid-2000s, my band of misfit UT Austin grad students and I had abandoned the attempt to program serendipity — not because the idea lacked merit, but because the web itself had made it unnecessary. Internet search, in its emergent form, already provided workarounds: we could rejigger keywords (“it’s something like…”), frequent discovery-oriented platforms like the now-defunct StumbleUpon, or, later, rely on the social graph. Once Facebook took off, our “friends” became serendipity engines of their own, feeding us surprises all day long.

Building serendipity into search became a nonstarter.

The loss of the unexpected

Image Credit: adistock - Adobe Stock

Even so, the web has never been the ideal medium for pure serendipity. I still find that wandering through old bookstores does a better job of summoning the angels (or sisters, if you know the story) of serendipity than anything algorithmic. This is part of a larger cultural turn.

Serendipity is not an isolated phenomenon — it belongs to a broader category of how discovery happens. This takes us to creativity.

From serendipity to creativity

Serendipity is a natural lead-in to creativity because both involve a departure from rule-following. If you find something you thought you didn’t want, then the rule that led you there —by definition — failed.

Rules, despite the old saying, aren’t made to be broken. They are made to deliver consistent results.

“If the rule you followed brought you to this, of what use was the rule? – Anton Chigurh, No Country For Old Men

The cultural decline and slow death of serendipity

Serendipity survives in attenuated form on the web but, like creativity, has undergone a kind of reduction to the status quo, part of a larger theme in modern culture to demystify and disenchant concepts that, well, don’t fit mechanical or rule-based descriptions. The culture seems to have anticipated its left hemisphere command-and-control leaders here, as no one seems to have the time anymore to go looking for the latest news about DOGE and end up with a cool piece on the intelligence of the octopus.

As the mind is increasingly reframed in digital terms, serendipity — unless it can be measured, categorized, or controlled — is treated as an inefficiency. The public, too, seems increasingly content to let “thinking outside the box” mean assembling a slick PowerPoint rather than pursuing anything that might actually change their thinking.

A similar reduction has taken place with our concept of creativity. Not only has it been increasingly misunderstood, but its essential conditions—those that allow it to emerge at all—are being eroded. Iain McGilchrist (following many others) explains that creativity unfolds in three stages.

The three phases of creativity

McGilchrist describes creativity as unfolding in three essential phases, each with its own requirements—what must happen, what must not happen, and what can happen if an idea is carried through to completion.

1. Generation (generative requirements)

Image Credit: Negro Elkha - Adobe Stock

The first phase is generation — the chaotic, undirected process of preparing for and allowing ideas to germinate. This is not the same as “brainstorming,” which implies a deliberate, conscious search for candidate solutions — listing possibilities on a whiteboard, for instance. True creative generation is messier, nonlinear, and difficult to formalize.

Generative requirements are what must happen for creativity to emerge.

2. Permission (permissive requirements)

Next comes permission — a stage that involves stepping aside so that an idea or insight can surface. This stage is not rational in any strict sense; it is a mystery, even to those experiencing it. Crucially, in order to have any hope of success, one must not try. Ideas percolate subconsciously. Sleep becomes more important than study. Serendipity belongs here, as do other “unwillable” aspects of life.

Permissive requirements are what must not happen — over-efforting, rigid structuring, or forcing an idea too soon.

3. Translation (translational requirements)

Finally, there is translation — the phase where the raw insight is shaped into something usable. Kekulé, upon dreaming of a snake devouring its own tail, still had to translate that image into the benzene ring. The idea alone meant nothing without this step.

Translational requirements define what can happen — if ideas are carried through to completion. Here, a bit of aplomb and raw courage prove helpful, and the rational mind can finally play a role.

Creativity as a black box

While the translation phase can be studied (it is largely ex post facto), the generation and permission phases remain fundamentally opaque. We don’t know where ideas come from — generative requirements are never a sure bet. Nor do we fully understand why some ideas rise while others vanish — permissive requirements are mostly about avoiding “blocking” activities.

Creativity remains one of the last true black boxes of cognition — of mind itself, or perhaps of the universe. And yet, modern culture seems increasingly indifferent to this mystery. The conditions that allow creativity to emerge — the space for ambiguity, the freedom to let ideas percolate — are steadily being eroded. Our tech-driven society seems uniquely bad at preserving these conditions, even as it grows more fascinated with the mechanics of thought itself.

Lacking a way to nurture creativity, we have turned instead to studying it. What we can do today, increasingly, is observe creative minds in action — glimpsing the process through neuroscience.

This isn’t QED. But it’s better than business books. Let’s turn to neuroscience next.

Next: Stranger Things: Why Mad Scientists Are Mad