ChatGPT and other forms of generative AI certainly seem impressive. The ability to produce grammatically correct text, workable code, and beautiful pictures has captivated the world and caused some economists to claim they have found the means of accelerating productivity.

A Federal Reserve Board governor says she is impressed. Lisa Cook’s speech on September 22, 2023, at the National Bureau of Economic Research Economics of Artificial Intelligence Conference led Bloomberg to publish an article titled “Fed’s Cook Sees Signs of AI Improving US Labor Productivity.” Yet the speech contains no evidence that this is occurring, being filled instead with statements such as,

“I am optimistic about broad benefits accruing to the economy and society from the use of generative AI — including more productive and less tedious work in offices, labs, factories, and warehouses — provided we address the very real concerns I just mentioned, and others like them.”

Equally important, if we look at the papers presented at the conference, we also can’t find evidence that AI is or will improve the productivity of factories, farms, mines, construction sites, hospitals, or retail outlets. One paper contrasts the growth-generating potential of AI vs. its existential risks without ever demonstrating any type of growth potential. A second paper examines hypothesis generation for the social sciences by looking at a judge’s decisions about who to jail. A third paper estimates the percentage of jobs that will be impacted by ChatGPT, the same approach used to estimate the impact of robots on worker tasks in many studies over the last 10 years, impacts that never materialized. A fourth paper examines Zillow’s home flipping model.

Looking outside the conference we find some analyses that demonstrate improvements in productivity. For instance. MIT researchers found that access to AI assistance increases the productivity of customer service workers in the Philippines “by 14 percent, as measured by the number of customer issues they are able to resolve per hour.” But these “gains accrue disproportionately to less-experienced and lower-skill workers,” preventing us from concluding that American workers, particularly knowledge workers, will benefit from this type of implementation.

A paper by Harvard Business School and Wharton professors, and Boston Consulting Group consultants, Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality,  does look at knowledge workers, in particular, strategy consultants. Using the consultants within BCG as guinea pigs, one group was asked to “conceptualize and develop new product ideas, focusing on aspects such as creativity, analytical skills, persuasiveness and writing skills while the other half (373 consultants) engaged in business problem-solving tasks using quantitative data, customer and company interviews,” and writing.

The problem with this approach is that more productive strategy consultants may not lead to higher productivity in their clients, and it is the client’s productivity that is the issue. It would be like arguing that the productivity of manufacturing engineers can be increased without considering the productivity of the factories in which they work. Nobel Prize-winning economist Robert Shiller once said, “Finance is not about making money per se . . . it exists to support other goals—those of society.” The same can be said of consultants and other knowledge workers. Whether they can be considered productive or not depends on whether their advice makes someone else more productive, and there is little evidence that strategy consultants are doing this for America’s economy.  

If they were, we would expect that the rapid growth in strategic consulting over the last fifty years would have caused an acceleration in productivity growth, but instead, a slowdown has emerged, something that Robert Gordon demonstrated in his book The Rise and Fall of American Growth. Although we doubt that strategy consultants are responsible for the slowdown, analyses of their work (Mariana Mazzucato and Rosie Collington, The Big Con: How the Consulting Industry Weakens Our Businesses, Infantilizes Our Governments, and Warps Our Economies) suggest the impact of strategy consultants on national productivity is mixed at best.   

Coding may receive the largest benefit from generative AI, but even here it will take time.

According to the Wall Street Journal, many corporate IT leaders are worried that “lowering the barrier for code creation could also result in growing levels of complexity, technical debt and confusion as they try to manage a ballooning pile of software. Technical debt is a broad term describing the expected future costs for applying quick-fix solutions.” The IEEE Spectrum also expresses caution in “Coding with ChatGPT, GitHub Copilot, and other AI tools is both irresistible and dangerous.” Many organizations are moving slowly with generative AI for coding and other work because of these types of problems.   

I am optimistic about AI in the long run, believing that it will impact positively on productivity. But it will take time. Economists should focus more of their analyses on processes than on individual tasks, because processes have a bigger impact on economic productivity than do tasks, particularly those such as writing emails, letters of recommendation, or reports, because someone must read those words, a fact not considered in studies of writing with ChatGPT. Real productivity comes from more useful ideas being expressed in words than before, and only downstream customers can make a legitimate judgment about usefulness. The customer service agents mentioned above were more productive because they were able to resolve a problem faster than before, as defined by the customer. We need more studies like this, ones in which real customers define the quality of the output. Even for coding, corporate productivity is more about how well code fits together than how long each bit of it took to code, and only customers can determine this.