Mind Matters Natural and Artificial Intelligence News and Analysis
college campus
Cherry trees in full bloom at the University of Washington campus
Image licensed via Adobe Stock

Goodhart’s Law and Scientific Innovation in Academia

Many university researchers are leaving academia so they can actually get things done

British economist Charles Goodhart was a financial advisor to the Bank of England from 1968 to 1985, a period during which many economists (“monetarists”) believed that central banks should ignore unemployment and interest rates. Instead, they believed that central banks should focus on maintaining a steady rate of growth of the money supply. The core idea was that central banks could ignore economic booms and busts because they are short-lived and self-correcting (Ha! Ha!) and should, instead, keep some measure of the money supply growing at a constant rate in order to keep the rate of inflation low and constant. The choice of which money supply to target was based on how closely it was statistically correlated with GDP.

The British monetary authorities adopted this policy in 1971 but were soon disappointed. In 1975 Goodhart bemoaned that, “The subsequent experience of trying to operate this revised system has, however, been troublesome and at times unhappy.” His cynical explanation was that “whenever a government seeks to rely on a previously observed statistical regularity for control purposes, that regularity will collapse.” When the central bank tried to stabilize a money supply measure that had been correlated with GDP, it ceased to be correlated with GDP. Goodhart’s observation has evolved into a more general rule that is now called Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure.”

A prime example is in academia where achievement is measured by the number of published research papers. Because the number of papers is the target, professors find ways to publish lots (and we mean lots) of papers, most of which are of little or no value. The unintended consequence is that instead of doing worthwhile, but time-consuming, research that might result in a small number of important papers, researchers waste their time writing a large number of inconsequential papers.

Publish or Perish

The cruel slogan “publish-or-perish” is a brutal fact of life. Promotions, funding, and fame all hinge on publications that purportedly demonstrate that a researcher is worthy of being promoted, supported, and celebrated. Anirban Maitra, a physician and scientific director at MD Anderson Cancer Center, wryly observed that “Everyone recognizes it’s a hamster-in-a-wheel situation, and we are all hamsters.”

The h-index (named after its creator Jorge E. Hirsch) is equal to the number of articles (h) that have been cited at least h times and is widely used for hiring, promotion, and funding decisions. The h-indexes are relatively small for many scientific giants who published a small number of extraordinary books and papers. The h-indexes are 97 for Charles Darwin, 70 for Niels Bohr, and 59 for Richard Feynman. Thousands of lesser scientists today (5,882 by a recent count) have an h-index of at least 100 and their ranks are increasing rapidly. When a measure becomes a target, …

A recent Nature article identified more than 9,000 individuals who had published 72 or more papers in a single calendar year. That’s an average of one published paper every five days!

We know why they do it (Goodhart’s law), but how? We don’t know the details of these particular bountiful authors but we do know some strategies that many people use. One is to develop a network of researchers who list each other as co-authors. I put your name on my paper and you put my name on your paper. The consequences can be absurd. Between 2014 and 2018, 1,315 papers were published that had 1,000 or more co-authors. A 2004 paper had 2,500 co-authors, a 2008 paper had 3,100; and a 2015 paper had 5,154 (it took 24 of the paper’s 33 pages to list all the co-authors). A 2021 paper set the Guinness record with 15,025 co-authors and celebrated this dubious achievement on Twitter.

From SCIGen to GPT-3

A more odious strategy is to automate the process of generating papers. In 2005 three MIT graduate students created a prank program called SCIgen that uses randomly selected words inserted into grammatically correct sentences to write bogus computer-science papers. They have now gone on to bigger and better things, but SCIgen lives on. Some researchers have bolstered their CVs by using SCIgen to create papers out of nothing. Cyril Labbé, a computer scientist at the University of Grenoble, wrote a program to detect hoax papers published in real journals. Working with Guillaume Cabanac at the University of Toulouse, they found 243 bogus papers published in 19 reputable journals.

Now we have GPT-3 and other large language models (LLMs) which can crank out papers faster than any human and which, unlike SCIgen, are consistently coherent and persuasive, though not always reliable. Just last week, Nature reported that GPT-3 is already being included as a co-author on some published papers. That is more forthright than hiding authorship (which will surely happen) but still eye-popping.

This publication pursuit is a waste of time for intelligent, educated, and energetic researchers. Another recent Nature article argued that, while there has been an explosion in the number of resources devoted to scientific research and published scientific papers, there has been a substantial decline over the past 75 years in genuine innovation (what they call “disruptive” science and technology). One of the authors, Michael Park, lamented, “We should be in a golden age of new discoveries and innovations.”

Leaving Academia to Get Things Done

Yet another recent Nature article provided evidence, mostly anecdotal, that many researchers are leaving academia for industry because they want to escape the hamster wheel, enjoy their work, and perhaps do some good. An epidemiologist and public health researcher said that the “genuine teamwork among colleagues all working together toward a shared goal is wonderful.” A cardiologist concurred: “We had this opportunity to literally rewrite the genome for [heart] health…. This could not have been done in academia, by any stretch of the imagination.”

These researchers are not the first to leave universities, and they will not be the last. Katlin Kariko spent nearly four decades struggling in academia while she researched messenger RNA (mRNA), which led to safe and effective COVID-19 vaccines. After she received the 2022 Breakthrough Prize in Life Sciences she said, “When I went from academia to a company, they don’t care how many committees you are on, how many papers you have. What counts is that you have a product that has an effect. The ego is wiped out. It is so much better.” Many other breakthroughs have been developed at corporate laboratories. For instance, looking at the Physics and Chemistry prizes awarded since 2000 in Li-ion batteries, LEDs, charge-coupled devices, double-heterostructure and quantum well lasers, giant magnetoresistance, integrated circuits, and optical fiber, 9 of the 17 recipients did their work at corporate labs.

The fundamental problem that diverts university research and stifles innovation is Goodhart’s law. When scientific innovation is measured by publication counts, we will get more publications and less innovation.

Gary N. Smith

Senior Fellow, Walter Bradley Center for Natural and Artificial Intelligence
Gary N. Smith is the Fletcher Jones Professor of Economics at Pomona College. His research on financial markets statistical reasoning, and artificial intelligence, often involves stock market anomalies, statistical fallacies, and the misuse of data have been widely cited. He is the author of dozens of research articles and 16 books, most recently, The Power of Modern Value Investing: Beyond Indexing, Algos, and Alpha, co-authored with Margaret Smith (Palgrave Macmillan, 2023).

Jeffrey Funk

Fellow, Walter Bradley Center for Natural and Artificial Intelligence
Jeff Funk is a retired professor and a Fellow of Discovery Institute’s Walter Bradley Center for Natural and Artificial Intelligence. His book, Competing in the Age of Bubbles, is forthcoming from Harriman House.

Goodhart’s Law and Scientific Innovation in Academia