Have you ever been in an online discussion where a vocal proponent confidently claimed that his opponent was the victim of the dreaded “Dunning–Kruger” effect? At Vox, Brian Resnick explains, “That’s where people of low ability — let’s say, those who fail to answer logic puzzles correctly — tend to unduly overestimate their abilities”:

An obvious example people have been using lately to describe the Dunning-Kruger effect is President Donald Trump, whose confidence and bluster never wavers, despite his weak interest in and understanding of policy matters. But you don’t need to look to Trump to find an example of the Dunning-Kruger effect. You don’t even need to look at cable news.

Brian Resnick, “An expert on human blind spots gives advice on how to think” at Vox (June 26, 2019)

Resnick quoted from his interview earlier with David Dunning, the University of Michigan psychology professor who contributed half of the name of the effect, emphasizing the importance of the intellectual humility we should all cultivate in order to avoid being the sort of low-ability people who overestimate ourselves.

Psychology Today’s glossary likewise warns,

The Dunning-Kruger effect is a cognitive bias in which people wrongly overestimate their knowledge or ability in a specific area. This tends to occur because a lack of self-awareness prevents them from accurately assessing their own skills.

The dreaded assumed effect derives its fame from a 1999 paper by David Dunning and Justin Kruger which received nearly 4600 citations: “The pair tested participants on their logic, grammar, and sense of humor, and found that those who performed in the bottom quartile rated their skills far above average. For example, those in the 12th percentile self-rated their expertise to be, on average, in the 62nd percentile.” – Psychology Today

So if you think that the online bully is mistaken, you better think again.

But by 2020, some observers began to suggest that the Dunning–Kruger effect isn’t real, much as they might like it to be:

I want the Dunning-Kruger effect to be real. First described in a seminal 1999 paper by David Dunning and Justin Kruger, this effect has been the darling of journalists who want to explain why dumb people don’t know they’re dumb.

But as I double-checked the academic literature, doubt started to creep in.

Jonathan Jarry M.Sc., “The Dunning-Kruger Effect Is Probably Not Real” at McGill Office for Science and Society (December 17, 2020)

Biologist Jarry started talking to statisticians and a different picture emerged:

In 2016 and 2017, two papers were published in a mathematics journal called Numeracy. In them, the authors argued that the Dunning-Kruger effect was a mirage. And I tend to agree.

The two papers, by Dr. Ed Nuhfer and colleagues, argued that the Dunning-Kruger effect could be replicated by using random data. “We all then believed the [1999] paper was valid,” Dr. Nuhfer told me via email. “The reasoning and argument just made so much sense. We never set out to disprove it; we were even fans of that paper.” In Dr. Nuhfer’s own papers, which used both computer-generated data and results from actual people undergoing a science literacy test, his team disproved the claim that most people that are unskilled are unaware of it (“a small number are: we saw about 5-6% that fit that in our data”) and instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.

Jonathan Jarry M.Sc., “The Dunning-Kruger Effect Is Probably Not Real” at McGill Office for Science and Society (December 17, 2020)

Popular culture continues to believe:

When I plug “Dunning-Kruger effect” into Google News, I get over 8,500 hits from media outlets like The New York Times, New Scientist, and the CBC. So many simply endorse the effect as a real bias of the brain, so it’s no wonder that people are not aware of the academic criticism that has existed since the effect was first published.

Jonathan Jarry M.Sc., “The Dunning-Kruger Effect Is Probably Not Real” at McGill Office for Science and Society (December 17, 2020)

Is the effect too good and the putdown too easy to pass up?

Late last year, Alexander Danvers, who researches emotions and social interactions at the University of Arizona, attempted a careful assessment of reasons for belief and doubt:

First, let’s consider how we came to believe in the Dunning-Kruger effect. Classic studies on the topic ask people to judge how well they would perform on various tests of intelligence and social skills, and then compares their actual performance on those tests to their peers. It turns out that people who did the worst estimate their performance to be much higher, relative to their performance, than others. In fact, this over-estimate of how well they did gets smaller and smaller the better the person actually did. This suggested to psychologists Dunning and Kruger that people who don’t know much are more overconfident than those who know a lot.

Alexander Danvers, “Dunning-Kruger Isn’t Real: The least knowledgeable people are not the most overconfident.” at Psychology Today (December 30, 2020)

While that seemed like a reasonable interpretation of the data, scientists also use a results check called a “model” that is not usually attempted in psychology, Danvers says: A model is a “simplified universe” that can be used to generate data on mathematical principles, based on our description of what we think is happening. “If the fake data we got from the model looks like the real data we see when we actually measure people, we can have some confidence that the model is right.”

But that wasn’t what happened when George Mason University psychologist Patrick McKnight, a statistician contacted by Jarry, tried replicating Dunning and Kruger’s data:

The controversy stirred up around Dunning-Kruger by the recent blog post was based on McKnight finding that he could create something that looked a lot like the Dunning-Kruger effect from a model where the worst performing people weren’t any more or less wrong about their skill level. People were just wrong randomly, and the pattern looked similar to the one originally published by Dunning and Kruger. A refinement of this was then posted by Benjamin Vincent of the University of Dundee in Scotland. In Vincent’s version, people were biased, but there was no difference between those who know the most and those who know the least. Everyone was just a bit overconfident in their abilities, no matter what level they were at. This matched the observed data beautifully.

Alexander Danvers, “Dunning-Kruger Isn’t Real: The least knowledgeable people are not the most overconfident.” at Psychology Today (December 30, 2020)

It also matches common sense. Most of us are too subjective most of the time to estimate our abilities accurately.

Jarry offer sympathy to the psychologists struggling with these types of measures:

This story is not over. There will undoubtedly be more ink spilled in academic journals over this issue, which is a healthy part of scientific research after all. Studying protons and electrons is relatively easy as these particles don’t have a mind of their own; studying human psychology, by comparison, is much harder because the number of variables being juggled is incredibly high. It is thus really easy for findings in psychology to appear real when they are not.

Jonathan Jarry M.Sc., “The Dunning-Kruger Effect Is Probably Not Real” at McGill Office for Science and Society (December 17, 2020)

You may also wish to read: Back to school briefing: Seven myths of social psychology: Many lecture room icons from decades past are looking tarnished now. (That was 2014 and devout belief in shakable truths continues in many places.)