Mind Matters Natural and Artificial Intelligence News and Analysis

TagStatistical significance


The Brady Bunch – Why Research Should Be Guided By Common Sense

Do our names really influence our choice in profession or the way our lives play out?

The credibility of scientific research is undermined by scientists torturing and mining data in a tenacious search for media-friendly results. Media-friendly findings tend to be entertaining, provocative, and surprising, and there is a good reason why they are surprising – they are wrong. Here is an example from BMJ, a top-tier medical journal. A paper with the alluring title, “The Brady Bunch?,” investigated “nominative determinism,” the idea that our surnames influence our choice of professions. With my name being Smith, I might have been predestined to choose to be a blacksmith or silversmith. That didn’t happen, but a newspaper article did find “a dermatologist called Rash, a rheumatologist named Knee, and a psychiatrist named Couch.” The authors of the BMJ…

Decreasing and increasing graph on chalkboard

The Cult of Statistical Significance — and the Neglect of Oomph

Statistical significance has little meaning when separated from practical importance

A central part of John Maynard Keynes’ explanation of the Great Depression was his assertion that household income affects household spending. When people lose their jobs and income, they cut back on their spending, which causes other people to lose their jobs and their income — propelling the economy downhill. Keynes’ theory was based on logic and common sense. It was later tested empirically with household survey data and with national income data compiled by the National Bureau of Economic Research. Figure 1 shows U.S. after-tax personal income and consumer spending for the years 1929 through 1940. Since income and spending both tend to grow over time along with the population, the data were converted to annual percentage changes. The observed statistical…

Worried doctors and medical researchers on conference meeting, discussing possible solutions for resolving a world health crisis. Health and medical care concept. Selective focus.

Computers Excel at Finding Temporary Patterns

Which contributes to the replication crisis in science

The scientific method calls for the rigorous testing of plausible theories, ideally through randomized controlled trials. For example, a study of a COVID-19 vaccine might give the vaccine to 10,000 randomly selected people and a placebo to another 10,000, and compare the infection rates for the two groups. If the difference in the infection rates is too improbable to be explained by chance, then the difference is deemed statistically significant. How improbable? In the 1920s, the great British statistician, Sir Ronald Fisher , said that he favored a 5 percent threshold. So 5 percent became the Holy Grail. Unfortunately, the establishment of a 5 percent hurdle for statistical significance has had the perverse effect of encouraging researchers to do whatever…