The credibility of scientific research is undermined by scientists torturing and mining data in a tenacious search for media-friendly results. Media-friendly findings tend to be entertaining, provocative, and surprising, and there is a good reason why they are surprising – they are wrong.
Here is an example from BMJ, a top-tier medical journal. A paper with the alluring title, “The Brady Bunch?,” investigated “nominative determinism,” the idea that our surnames influence our choice of professions. With my name being Smith, I might have been predestined to choose to be a blacksmith or silversmith. That didn’t happen, but a newspaper article did find “a dermatologist called Rash, a rheumatologist named Knee, and a psychiatrist named Couch.” The authors of the BMJ paper add their own examples:
Was it a surprise that during the London 2012 Olympics, a man who shares his name with lightning became the fastest man in the world? Usain Bolt ran the 100 metre final in 9.63 seconds, faster than anyone else in history. It is also somewhat predictable that Bulgarian hurdler Vania Stambolova would, well, stumble over as she did, unfortunately falling at the first hurdle in her heat.John J Keaney et al., “The Brady Bunch? New evidence for nominative determinism in patients’ health: retrospective, population based cohort study” at The BMJ
Not every dermatologist is named Rash nor is every runner named Bolt. Of all the many people in any profession, there are bound to be some whose names are coincidentally related to their profession. The question is whether this is true for an unusually large number of people.
The authors focused on whether Dubliners named Brady have an unusually large incidence of bradycardia. Now, unless you are a doctor, you are probably wondering (as I was) what bradycardia is. The word bradycardia comes from the Greek words bradys (“slow”), and kardia (“heart”), and signifies a slower than normal heart rate which may be treated with the surgical implantation of a pacemaker.
The argument that people named Rash might be attracted to professions dealing with rashes assumes, reasonably enough, that they know what the word rash means. The fact that bradycardia is an unfamiliar word for the overwhelming majority of the population demolishes the idea that people named Brady would be psychologically susceptible to bradycardia.
Nonetheless, the authors looked at 999 pacemaker implants at an unnamed hospital in Dublin during the period January 2007 through February 2013, and found that 8 (0.80 percent) were implanted in patients named Brady. For comparison, they looked at 161,967 residential telephone listings in the area of the hospital and found that 579 (0.36 percent) were named Brady. The probability of such a large disparity is 0.03, which is (marginally) statistically significant by Sir Ronald Fisher’s 5% rule.
Given the utter implausibility of the hypothesis, how did the authors find a statistically significant pattern? There are several ways in which the data may have been p-hacked, starting with the fact that the authors may have looked at several diseases in addition to bradycardia. Focusing on bradycardia, there are eight Dublin hospitals in which pacemaker implants are performed, and the authors are affiliated with three of them, yet results are only reported for a single hospital. Did they omit hospitals with contradictory data?
The authors say that the data were for a 61-month period from January 1, 2007, through February 28, 2013. This is actually a 74-month period, which raises questions about how carefully the calculations reported in the paper were carried out. Readers might also wonder why the time period is approximately 6 years, instead of the more obvious 5 years or 10 years? Also, why did the time period end on February 28, 2013, rather than December 31, 2012, which would have been exactly six years, or end in a month closer to the December 2013 publication date? Did January and February 2013 help the case they were trying to make while March, April, and May 2013 hurt the case?
The frequency with which the surname Brady appears in telephone directories is a noise measure of the number of Bradys who were candidates for pacemaker implants. For one thing, pacemaker recipients are somewhat older than the telephone directory population and perhaps the prevalence of the surname Brady has declined over time and is consequently more common among older citizens. Second, even though the directory is labeled residential, some entries are not households. My search for Ltd and Limited in the Dublin residential directory turned up dozens of unique non-household results (some listings appear multiple times), including:
Findings Ireland Ltd
DecathlonSports Ireland Ltd
Roofing Ltd Main Street
Anbawn Computing Limited
Services Limited BROC Accounting
I also stumbled across these listings:
National Council for the Blind
The Swedish Coffee Company
There is no easy way to determine how many of the residential Dublin listings are duplicates or not households, but it is clear that the correct false-positive probability is higher than the 0.03 value calculated by the authors and that the other problems with this paper make the statistical evidence unconvincing.
I’ve written before about the need to consider “oomph,” the practical importance of the findings. Is the difference between a 0.80 percent and 0.36 percent incidence of any real significance? Should people named Brady fear bradycardia? Should doctors test for bradycardia in everyone named Brady?
The goal of scientific research ought to be more than media-friendly findings that are entertaining, provocative, and surprising. Real science is not fake news.
You might also enjoy:
The Cult of Statistical Significance – and the Neglect of Oomph. Statistical significance has little meaning when separated from practical importance. A misplaced focus on statistical significance is an inherent weakness of deep neural networks and other black box models. (Gary Smith)