Matthew Walker is a professor of neuroscience and psychology and founder of the Center for Human Sleep Science at the University of California, Berkeley. He has become famous for his book and a TED talk promoting the importance of sleep for health and performance. He even got a job at Google as a “sleep scientist.”

Walker has a receptive audience because he is entertaining and his arguments make sense. In one of his books, Walker used a graph similar to the figure below to show that a study done by other researchers had found that adolescent athletes who sleep more are less likely to be injured.

The figure is compelling, but there are several potential problems. The hours-of-sleep data were based on 112 responses to an online survey of athletes at a combined middle school/high school. The injury data were from logs of students who came to the school athletic trainer’s room for “evaluation and/or treatment.” Overall, 64 the 112 athletes made a total of 205 visits.

Online surveys are notoriously suspect and recollections of the average hours of sleep are likely to be unreliable. The training room data were over a 21-month period but, nonetheless, seem high. Perhaps some middle school/high school students preferred the trainer’s room to being at practice.

The most serious problem, however, is not that Walker was relying on flimsy data. Finnish fitness blogger Olli Haataja took the trouble to read the original study, perhaps because he was interested in the sample size or the age of the athletes. He discovered that the graph below, which was reported in the original study, showed five sleep categories, instead of the four Walker reported. Walker had omitted the 5-hour category, which contradicted his argument! It is difficult to explain the omission as anything other than a deliberate attempt to misrepresent the results of the study.

Independent researcher Alexey Guzey put this falsification on a website he maintains listing errors in Walker’s work. Andrew Gelman, a professor of statistics and political science at Columbia University, then wrote about Walker’s distortion in a blog and in an article co-authored with Guzey, arguing that, “When it comes to unethical statistical moves, hiding data is about as bad as it gets.”

A software developer named Yngve Hoiseth was so incensed that he contacted the University of California to report that Walker had been guilty of research misconduct, as defined by the University:

Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.

Berkeley Research, “Research Misconduct”

Walker clearly misrepresented the research record by omitting some results.

The University staunchly defended its celebrity professor:

Thank you for your interest in this matter, which we have pursued in accordance with our policy. In conversation with Walker and with the professor who conducted the inquiry, the conclusion was that the bar omitted from the graph on the book did not alter the findings in an appreciable way, and more importantly, that the bar was not omitted in order to alter the research results.

As quoted in: Yngve Hoiseth, “Why We Sleep: a tale of institutional failure,” March 24, 2020

Gelman subsequently asked the obvious question: “If the removal of the bar from the graph didn’t matter, then why did you remove the damn bar?”

This misrepresentation of results does not seem to be an isolated incident. Markus Loecher, a professor of mathematics and statistics at the Berlin School of Economics and Law, reported similar mischief in Walker’s TED talk, “Sleep is your superpower.” At one point, Walker made this dramatic argument:

I could tell you about sleep loss and your cardiovascular system, and that all it takes is one hour. Because there is a global experiment performed on 1.6 billion people across 70 countries twice a year, and it’s called daylight saving time. Now, in the spring, when we lose one hour of sleep, we see a subsequent 24-percent increase in heart attacks that following day. In the autumn, when we gain an hour of sleep, we see a 21-percent reduction in heart attacks. Isn’t that incredible? And you see exactly the same profile for car crashes, road traffic accidents, even suicide rates.

Matt Walker, “Sleep is your superpower,” TED Talk on YouTube

After listening to this, Loecher wrote:

Now I tend to be sensitive to gross exaggerations disguised as “scientific findings” and upon hearing of such a ridiculously large effect of a one-day-one-hour sleep disturbance, all of my alarm bells went up!

He contacted Walker and was told that the source of these claims was a study of Michigan heart attacks following four spring and three fall daylight savings time changes, not “a global experiment performed on 1.6 billion people across 70 countries.”

The table below shows that there was indeed a 24 percent increase after the spring change and a 21 percent decrease after the fall change, as Walker stated, but these fluctuations did not happen, as Walker claimed, the “following day.” Saturday night is when we have one less or one more hour of sleep and the following day is Sunday. The spring increase in the Michigan data was on a Monday and the fall decrease was on a Tuesday, two seemingly random days — except for the fact that they were the only days to have had p-values (0.011 an 0.044, respectively) below 0.05. The Sunday after the time changes, the number of heart attacks went in the opposite direction. There were also bulges and declines on other days. Even if the day of the week didn’t matter, some days will inevitably, by chance, have more or fewer heart attacks than others.

But perhaps, you will say, the sleep effects are felt during the entire week following the time changes. Nope. The Michigan authors clearly state that, “There was no difference in the total weekly number of [heart attacks] for either the fall or spring time changes.” The authors also caution that they had used multiple statistical procedures and that,

No adjustments were made for multiple comparisons, and the analysis is intended to be exploratory in nature. As such, nominally significant results should be interpreted as hypothesis generating, rather than confirmatory evidence.

Amneet Sandhu, Milan Seth, and Hitinder S Gurm, “Daylight savings time and myocardial infarction” at NCBI

Relative Risk of Heart Attack During the Week After Daylight Saving Time Changes

Walker took the puzzling and perhaps coincidental results of a small exploratory study and inflated it into a claim that were well-documented worldwide surges and declines in heart attacks on the Sunday following time changes.

Loecher also reported that, unsurprisingly, the puzzling Michigan results did not hold up in other data. He contacted one of the authors of the Michigan study and was told that the authors “looked for the same signal in more recent data and it is markedly attenuated and no longer significant.” Loecher also reported — and this, too, should come as no surprise — that, “I was unable to find any backing of the statement on ‘exactly the same profile for car crashes, road traffic accidents, even suicide rates’ in the literature.”

Boring results do not receive worldwide publicity, TED talk invitations, or Google jobs — which is why it is tempting to misrepresent and exaggerate in order to impress and entertain. The social cost is that misrepresentations and exaggerations undermine the credibility of scientists by making them seem more like snake-oil peddlers than dispassionate scientists.