I wrote here yesterday about the perils of presidential election models that are p-hacked to make near-perfect “predictions” of elections that have already happened, using Helmut Norpoth’s Primary Model as an example. Another well-publicized model is Thirteen Keys to the White House, created in 1984 by Alan Lichtman, a history professor at American University.

Lichtman’s model uses 13 true/false questions (“keys”) that reflect sensible political and socioeconomic issues that affect voter enthusiasm for the incumbent party:

1. Party Mandate: After the midterm elections, the incumbent party holds more seats in the U.S. House of Representatives than after the previous midterm elections.
2. No Primary Contest: There is no serious contest for the incumbent party nomination.
3. Incumbency: The incumbent party candidate is the sitting president.
4. No Third Party: There is no significant third party or independent campaign.
5. Short-term economy: The economy is not in recession during the election campaign.
6. Long-term economy: Real per capita economic growth during the term equals or exceeds mean growth during the previous two terms.
7. Major Policy Change: The incumbent administration affects major changes in national policy.
8. No Social unrest: There is no sustained social unrest during the term.
9. No Scandal: The incumbent administration is untainted by major scandal.
10. No Foreign or Military failure: The incumbent administration suffers no major failure in foreign or military affairs.
11. Major Foreign or Military Success: The incumbent administration achieves a major success in foreign or military affairs.
12. Charismatic Incumbent: The incumbent party candidate is charismatic or a national hero.
13. Uncharismatic Challenger: The challenging party candidate is not charismatic or a national hero.

Interestingly, only two of the keys (12 and 13) depend on the party nominees: charismatic incumbent and uncharismatic challenger,

When eight or more of the 13 keys hold true, the incumbent party is predicted to win the popular vote. Six or more false keys and the opposition party is predicted to win.

Predicting vs. explaining

Lichtman became a celebrity presidential predictor in 2016 when he was virtually alone in predicting that Donald Trump would defeat Hillary Clinton. Even Trump seemed surprised when he won, as he had not prepared a victory speech and was planning to fly to Scotland to play golf.

One problem with Lichtman’s model is what statisticians call overfitting—using so many variables to explain so few presidential elections. Notice that I used the word explain instead of predict because the Keys model was designed to “predict” the winners of elections that have already occurred.

Evidence of p-hacking

A second problem is that Lichtman has p-hacked his model over time to better explain the past.

In an October 6, 2024 interview, Lichtman said that “if I tried to fiddle with my model in response to what people say are contemporary changes, I would just make errors. The keys are indeed the northern star of prediction. They don’t change.” The keys have, in fact, changed over time; no doubt, these changes are p-hacks that improve the “predictions” of past elections.

Lichtman’s original model, released in 1981, had only 12 keys. The current model dropped one key (“Has the incumbent party been in office more than a single term?”) and added two foreign policy/military keys (10 and 11). It also changed Key 1 from “Did the incumbent party gain more than 50% of the vote cast in the previous election?” The large number of changes after a handful of election results — three of which (1964, 1972, and 1984) were easily predicted landslides — is a sure sign that the model does better predicting the past than forecasting the future.

A third problem is that the model has considerable wiggle-room in that some of the true/false questions are quite subjective. That allows Lichtman to adjust his predictions as needed. For example, Lichtman waited an unusually long time to make his 2020 prediction on the contest between incumbent Donald Trump and challenger Joe Biden. The pollsters overwhelmingly predicted a Biden victory and Lichtman finally did too, stating that only six keys were true, two short of the required eight. But consider these two keys which Lichtman counted as false:

Foreign/Military Success, because of the lack of an acclaimed success abroad.

Incumbent Charisma, because Trump appeals only to a narrow base.

Lichtman counted Foreign/Military Success as false even though Trump was nominated for a Nobel Peace Prize for his work in facilitating the Abraham Accords, a peace agreement between Israel and the United Arab Emirates. Was that achievement worthy of a true checkmark? Who knows? That’s the problem.

The Incumbent Charisma key is even more problematic. Who could seriously deny that Trump has charisma? The rationalization that “Trump appeals only to a narrow base” sounds suspiciously circular: Trump appeals to a minority of voters, so I predict that he will lose.

This flexibility is particularly striking in that the Keys model has correctly predicted the outcome of 9 of the 10 presidential elections, 1984–2020, which is exactly the same record as political polls. Political polls may well influence how Lichtman answers subjective Keys.

Troubling inconsistencies

Lichtman’s inconsistencies are also troubling. He predicted that Al Gore would win the 2000 election and, when Bush won the electoral votes, Lichtman argued that his model explicitly predicts the winner of the popular vote, not the electoral vote — and Gore did win the popular vote.

Now back to the 2016 election, which made Lichtman’s reputation. The Keys model did not predict that Trump would win the electoral vote (which he did); it predicted he would win the popular vote (which he lost by nearly 3 million votes). Lichtman was very explicit about his prediction: writing in October 2016, shortly before the election, that “the Keys predict the popular vote, not the state-by-state tally of Electoral College votes.” Yet he continues to count his incorrect 2016 prediction as a correct prediction.

What’s Lichtman’s 2024 prediction? As the challenger, Trump needs six false keys and he only has four:

The mandate key because Democrats lost US House seats in 2022. Incumbency, it’s an open seat. And incumbent charisma, because Harris, whatever you may think of her, is certainly not yet a Franklin Roosevelt. And the Democrats also lose the foreign policy/military policy failure key for Gaza, which is a humanitarian disaster with no end in sight four.

Lichtman counts Key 7 (Major Policy Change) as true in “areas like the environment, immigration, infrastructure, climate change.” He also counts Key 11 (Major Foreign or Military Success) as true because “it was Biden whose support has helped keep Ukraine alive for over two and a half years.” And he continues to count Key 13 (Uncharismatic Challenger) as true because he doesn’t consider Trump charismatic. Flip any two of these three keys and Trump has the six keys needed for Lichtman’s model to predict Trump’s victory — though Lichtman remains unclear about whether he is predicting a victory in the popular or electoral vote. That’s something that will, no doubt, be clarified after the election.

You may also wish to read: Presidential Pundits — a p-hacking parable. In politics, as elsewhere, too many studies flop when other researchers attempt to replicate them with fresh data. Some prediction models were developed by well-intentioned researchers before the perils of p-hacking were clearly understood, hence the failures. (Gary Smith)