Peer review sleuths Leslie D. McIntosh and Cynthia Hudson Vitale contributed a paper recently to the voluminous literature on what’s wrong with peer review: “The objective of this study is to present a case study on how the peer review process may be manipulated by individuals with undisclosed connections to politically divisive organizations.” by which they mean pro-life ones:

This case study analyzes the expertise, potential conflicts of interest, and objectivity of editors, authors, and peer reviewers involved in a 2022 special journal issue on fertility, pregnancy, and mental health. Data were collected on qualifications, orga-nizational affiliations, and relationships among six papers’ authors, three guest editors, and twelve peer reviewers. Two articles were found to have undisclosed conflicts of interest between authors, an editor, and multiple peer reviewers affiliated with anti-abortion advocacy and lobbying groups, indicating compromised objectivity. This lack of transparency undermines the peer review process and enables biased research and disinformation proliferation.Our study is limited by a few factors including: difficulty collecting peer reviewer data, potentially missing affiliations, and a small sample with-out comparisons. While this is a case study of one special issue, we do have suggestions for increasing integrity.

McIntosh, L.D., & Hudson Vitale, C. (2023). Safeguarding scientific integrity: A case study in examining manipulation in the peer review process. Accountability in research, 1-19 .

They targeted a special Frontiers in Psychology issue (Sammut, S., P. Yeung, and D. Larrivee n.d. Fertility, Pregnancy and Mental Health – a Behavioral and Biomedical Perspective) and possibly brought about the retraction of one study: Coleman, P. 2022. when their research turned up multiple connections of authors with pro-life groups, which they identify as “conflict of interest” :

Our analysis found two of the three guest editors, five of the twelve (42%) peer reviewers, and two of nineteen authors (11%) are active in anti-abortion advocacy and lobbying organizations that are conflict of interests:(1)Charlotte Lozier Institute (CLI)(2)American Association of Pro-Life Obstetricians and Gynecologists (AAPLOG)(3)Breast Cancer Prevention Institute (BCPI)(4)World Expert Consortium for Abortion Research and Education (WECARE)(5)St Louis Guild of the Catholic Medical Association

McIntosh, L.D., & Hudson Vitale, C. Manipulation in the peer review process.

They conclude, “As disinformation grows more widespread, mitigating its encroachment into academic literature is crucial to protect society from its detrimental effects.”

Was the paper or the special issue good, bad, or indifferent? It hardly matters. We all know implicitly that heavy involvement in providing abortions at all stages of pregnancy would likely add to the sense, among many abortion-related peer reviewers, that a journal author was well qualified. And involvement in preventing abortions would detract from it. Thus, McIntosh and Hudson Vitale illustrate, by their very zeal, the built-in corruption of the peer review process. Reader Laszlo Bencze writes to say, “Interesting that anti-abortion advocacy might present a conflict of interest when reviewing scientific papers but pro-abortion advocacy would present no conflict. The reason for this is the pro-abortion is completely scientific whereas anti-abortion is clearly religious and hence anti-scientific. Interesting world we live in.”

And the academy blunders on

From Harvard, of all places, we learn, “Overall, there is a lot of work being done to improve upon the peer-review system. For example, journals are now putting in more effort to retract incorrect papers, so that papers that slip through the cracks are taken down in a more timely fashion, working to prevent the spread of false information.”

Doesn’t that fill us with confidence, given Harvard’s own recent history! See, for example, evolutionary biologist Carole Hooven, “Why I left Harvard”: ““After I stated banal facts about human biology, I found myself caught in a DEI web, without the support to do the job I loved. The only way out was to leave…” Harvard is going to have quite a job convincing the world that it is still serious about reality-based thinking, never mind peer review.

Goodhart’s Law is a tough opponent

Something our Walter Bradley Center director Robert J. Marks said about peer review in 2019 is worth a second look: Reformers — assuming they are even serious — are battling numerical laws that govern how incentives work.

Goodhart’s Law, for example, captures the unintended effect of using numerical metrics as goals: “When a measure becomes a target, it ceases to be a good measure.”

British economist Charles Goodhart’s law applies well beyond his specialty. As a general principle, focusing exclusively on a measurable numerical goal distracts from achieving the broader outcome that the metric is trying to track. Players focus on the numbers and try to game them, which often leads to deception and dishonesty. Crossing the finish line becomes more important than how you got there.

A consequence of Goodhart’s Law is the temptation to cheat to achieve a goal. This is called Campbell’s Law, after an American social scientist, Donald T. Campbell: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

In other words, measurement creates a temptation to achieve a measurable goal by less than totally honest means. As in physics, the simple act of measuring invariably disturbs what you are trying to measure.

Robert J. Marks, “Why It’s So Hard to Reform Peer Review,”Mind Matters News, July 18, 2019

That’s hard to battle even when we are trying to be fair-minded, which some probably aren’t.