Throughout my career, I have warned of the dangers of data mining—ransacking data for statistical relationships, buoyed by the misguided belief that correlation supersedes causation.

Data mining’s fatal flaw is that reasonably sized databases inevitably feature many statistical coincidences; finding a pattern only demonstrates that one looked for one. If there is a finite number of true causal relationships but an essentially infinite number of coincidental relationships, the chances that a discovered statistical pattern will be useful is close to zero.

This is why economics Nobel Laureate James Tobin advised that the bad old days when researchers had to do calculations by hand were actually a blessing. The calculations were so hard that people thought hard before calculating. Today, with terabytes of data and lightning-fast computers, it is too easy to calculate without thinking.

I was reminded of the perils of data mining by Google’s full-moon commercials during the current baseball playoffs. Gemini had discovered that,

• The number of multi-run late-inning comebacks increased by 34% under a full moon.
• The number of 7+ run blowout games increased by 25% under a full moon.
• The number of 450+ ft. home runs increased by 100% under a full moon

Each claim is semi-plausible if you believe that strange things happen during full moons, but each statistic is suspect because it was discovered by data mining. (Coincidentally, I once debunked a paper that had tortured a large database in order to reach the conclusion that fatal motorcycle accidents are more likely when there is a full moon.)

The harmful consequence of data mining is that the credibility of science and scientists is undermined when reported statistical patterns disappear because they were only fortuitous coincidences.

Why data mining is a problem

Data mining is also the source of the superstitious beliefs that many baseball players cling to, based on serendipitous events.

Using a rounded wooden stick to hit a small baseball traveling ninety-plus miles an hour and perhaps veering left, right, up, or down may be the most difficult challenge in any sport. If the ball is hit, it might go right to a fielder for an out or it might land safely for a base hit. On average, professional baseball players only get a base hit on one out of four times at bat. Increase that fraction to one out of three and you’re headed for the Hall of Fame. From the pitcher’s viewpoint, similar small differences separate all-star pitchers, journeymen, and failures.

Perhaps this is why baseball players are notoriously superstitious, looking for something—no matter how ridiculous—that might tilt the odds in their favor.

MLB players and their obsessions

Wade Boggs, for example, had an illustrious career playing third base, mostly with the Boston Red Sox. He had a career batting average of 0.328 (one out of three), played in twelve straight All-Star Games, and was elected to the Baseball Hall of Fame in 2005. If there was a Superstition Hall of Fame, he would be in that too. Boggs woke up at exactly the same time every day and ate chicken at 2:00 p.m. using a 13-recipe rotation over every 14 days (he ate lemon chicken twice).

For a night game at Fenway Park, the Chicken Man got to his locker at exactly 3:30, put on his uniform, and went to the dugout to warm-up at 4:00. He then went through a precise warm-up routine, including fielding exactly 150 ground balls. At the end of his fielding warm-up, he stepped on third base, second base, first base, and the baseline (during the game he entered the field by jumping over the baseline) and taking 2 steps to the coach’s box and 4 steps to the dugout. By the end of the season, Bogg’s footsteps had left permanent footprints in the grass. Boggs always took batting practice at 5:17 and ran wind sprints at 7:17 (an opposing manager once tried to disorient Boggs by having the stadium clock skip from 7:16 to 7:18).

During the game, when Boggs took his position at third base, he smoothed the dirt in front of him with his left foot, tapped his glove three times, and straightened his hat. Every time he batted, he drew the Hebrew word Chai (“life”) in the batter’s box. (He isn’t Jewish.)

As obsessive as Boggs was, he seems almost sane compared to Turk Wendell, who pitched in the major leagues for 11 years, from 1993 to 2004. He wore uniform 99 because this was Charlie Sheen’s uniform number when he played pitcher Ricky Vaughn (“Wild Thing”) in the movie Major League. When Wendell signed a contract with the New York Mets in 2000, he asked for a contract of $9,999,999.99. He wore a necklace made from the teeth of wild animals he had killed.

Before walking to the mound to pitch, Wendell put 4 pieces of black licorice in his mouth to chew on while he was pitching. Instead of stepping on the baseline (as some players do) or stepping over the baseline (as other players do), Wendell would leap over the baseline. When he reached the mound, Wendell waved to the center fielder until the player waved back. When he needed a new ball, he asked the umpire to roll it to him so he wouldn’t have to catch it; if the umpire threw the ball to him, Wendell would either let it bounce off his chest or let it go by him. After every inning, he brushed his teeth in the dugout, something so unusual that it is shown on his Upper Deck rookie baseball card.

Laugh (as you should) at this level of obsession. But, please, also laugh at Google’s Gemini commercials and at the claims of researchers who use data mining to find statistical patterns. There are far too many and AI makes it far too easy now.