Brain imaging has shed much light on medical conditions in recent decades. So it was hardly good news for neuroscientist Scott Marek at the University of Washington when the results of a study linking brain function with intelligence in 2000 children produced very counterintuitive results. He and his colleagues had divided the sample into two groups of 1000 and run the same analysis on each — and they did not match. At first, he told Nature, “I stared out of my apartment window in depression, taking in what it meant for the field.”

Then the team decided to study the problem on a larger scale, using the three key studies in this type of research, the Adolescent Brain Cognitive Development (ABCD) study, Human Connectome Project (HCP), and the UK Biobank, amounting to 50,000 scans. The results were sobering:

As a result, the conclusions of most published ‘brain-wide association studies’ — typically involving dozens to hundreds of participants — might be wrong. Such studies link variations in brain structure and activity to differences in cognitive ability, mental health and other behavioural traits. For instance, numerous studies have identified brain anatomy or activity patterns that, the studies say, can distinguish people who have been diagnosed with depression from those who have not. Studies also often seek biomarkers for behavioural traits.

“There’s a lot of investigators who have committed their careers to doing the kind of science that this paper says is basically junk,” says Russell Poldrack, a cognitive neuroscientist at Stanford University in California, who was one of the paper’s peer reviewers. “It really forces a rethink.”

Ewen Callaway, “Can brain scans reveal behaviour? Bombshell study says not yet” at Nature (March 17, 2022) The paper requires a fee or subscription.

The challenged field is not brain imaging as such but a specific subset of brain imaging — brain wide association studies (BWAS). The field attempts to account for behavior differences as differences in the brain. Such studies, often of psychiatric conditions, were the Next Step for neuroimaging: showing that human behavior is based on simple, identifiable brain states. But that’s not what happened:

The team found that, regardless of size, the types of association analyzed in BWAS studies were extremely prone to being inflated by chance. This meant that the findings of Marek’s smaller simulated studies were largely irreproducible. Only once the experiments were modeled with thousands of brain scans did the inflated effect sizes begin to reduce.

Essentially, small BWAS studies, set up to analyze minuscule potential effects with low numbers of scans, are highly likely to be irreproducible. Moreover, in these smaller studies, the authors write, it is the very findings that are most inflated by chance and least reliable that are most likely to be found “statistically significant” and make it to publication.

Ruairi J Mackenzie, “Stats Study Reveals Reason for Replication Crisis in Neuroscience” at Technology Networks Neuroscience News and Research (March 16, 2022)

Many BWAS studies take place in small labs on tight budgets with a median sample size of 23. But even the large ones aren’t nearly reliable enough to be a reasonable basis for decision-making:

Even associations identified in a study of 2,000 participants — large by current standards — had only a 25% chance of being replicated. More typical studies, with 500 or fewer participants, produced reliable associations around just 5% of the time.

The study did not attempt to replicate other published brain-wide association studies. But it suggests that high r values common in the literature are almost certainly a fluke, and not likely to be replicated. Factors that hinder reproducibility in other fields, such as the tendency to publish only statistically significant results with large effect sizes, means that these spurious brain–behaviour associations fill the literature, says Dosenbach. “People are only publishing things that have a strong enough effect size. You can find those, but those are the ones that are most wrong.”

Ewen Callaway, “Can brain scans reveal behaviour? Bombshell study says not yet” at Nature (March 17, 2022) The paper requires a fee or subscription.

In short, the Aha! news stories about what brain imaging reveals about human behavior are probably based on studies whose findings would not be confirmed by further research.

The researchers don’t claim that brain imaging is useless for studying behavior but that larger sample sizes are needed, which amounts to bigger, better funded studies. Funding sources, however, may want more certainty that the information would provide a clear, practical benefit.

You may also wish to read:

No, fMRI brain scans are NOT reading our minds. After reading her perceptive essay about the problems in fMRI imaging in neuroscience, I’m sad that a gifted student has doubts about a career in the field. Neuroscience badly needs skeptics to show how unreliable technology, biased handling of data, and materialism’s conceptual mess frustrate science. (Michael Egnor)