In a recent article at The Atlantic, King’s College psychologist Stuart Ritchie, author of Science Fictions: How Fraud, Bias, Negligence and Hype Undermine the Search for Truth (2020), has noted a curious fact: Video gamers are much quicker to spot fraud than scientists.

The video game fraud he focuses on involved a gamer’s claim that he had finished a round of Minecraft in a little over 19 minutes, a feat he attributed, Ritchie tells us, to “an incredible stretch of good luck.”

“Incredible” is the right choice of word here. “Dream,” as the player was known, later admitted — in the face of skepticism — that he had “inadvertently” left some software running that improved his game — thus disqualifying himself.

So far, the whole story really only matters to people who care about Minecraft. But Dr. Ritchie goes on to make a more sobering point for the rest of us.

Odd as we might find this, peer reviewed science publishing is actually less efficient at weeding out frauds than the gamers are. Just for example, gamers are expected to provide video proof of their hands on the keyboards:

The rigorous format in which the speedrunning community asks players to provide video proof of their runs is itself significant. For many games, you need to show not just a recording of your screen, but also a video of your hands on the controller or keyboard, so moderators can ensure that it really was a human—and not a script or a bot—that clinched the all-important record.

Science has been much slower to adapt, even after countless scandals. Researchers provide images for their papers entirely at their own discretion, and with no official oversight; when they aren’t faked, they might still contain cherry-picked snapshots of experiments that don’t represent the full range of their results. The same applies to numerical data, which are often—consciously or unconsciously—chosen or reported to make the best case for a scientist’s hypothesis, rather than to show the full and messy details. Only a few journals require scientists to do the equivalent of posting the screen-and-hands recording: sharing all their data, and the code they used to analyze it, online for anyone to access.

Stuart Ritchie, “Why Are Gamers So Much Better Than Scientists at Catching Fraud?” at The Atlantic (July 2, 2021)

The story is sobering reading, especially when we consider that many science papers are actually significant for health, welfare, and the environment.

You may also wish to read: Why It’s So Hard to Reform Peer Review/a> Reformers are battling numerical laws that govern how incentives work. Know your enemy!

and

Publish or Perish! Another example of Goodhart’s Law. In becoming a target, publication has ceased to be a good measure.