Mind Matters Natural and Artificial Intelligence News and Analysis
frozen-british-traditional-post-office-mailbox-stockpack-adobe-stock.jpg
Frozen British traditional post office mailbox

Who Will Your Boss Believe? You or That Glitzy New Computer?

In the largest miscarriage of justice in decades, the British Post Office chose to believe the computer, resulting in bankruptcies, jail, and suicide. At last the truth emerged…
Share
Facebook
Twitter
LinkedIn
Flipboard
Print
Email

Texas State professor of electrical engineering Karl Stephan (pictured) has the story, reprinted with permission from his blog Engineering Ethics. Dr. Stephan is the author of Ethical and Otherwise: Engineering in the Headlines, a collection of his writings on ethics and technology:

Suppose you enjoy a secure government job at which you work diligently, and you have advanced to the managerial position of a sub-postmaster in Post Office Ltd, the quasi-public organisation that provides postal services in most of the UK.  Then your organisation installs a new computerised system called Horizon that promises to eliminate a lot of paperwork accounting and make things easier for everybody.  But soon after it is installed, you find that your accounts are not matching up with what the computer says they are.  You bring these discrepancies to the attention of your supervisors, but instead of looking into the problem, they accuse you of stealing the deficit funds, amounting to many thousands of pounds in some cases.

Something like this happened to dozens of UK sub-postmasters over the last two decades.  Every time the computer indicated an unexplained deficit, the Post Office concluded that the sub-postmasters were responsible, and threatened many of them with prison terms if they didn’t make up the deficit personally.  Some did, mortgaging their houses and even going bankrupt, but others went to jail anyway.  The accusations of theft led to psychological problems, broken marriages, and at least one reported suicide.

Meanwhile, the Post Office authorised an outside agency called Second Sight to conduct an independent investigation after it failed on its own to find out what was really going on.  One day before Second Sight was to publish its report in 2015, the Post Office cancelled the investigation, ordered the agency to destroy its files, and issued a public statement denying that there were any systemic problems. 

Things went on like this until December of 2019, when the Post Office began to admit publicly that it was wrong in many of the cases.  And in March, the UK’s Court of Appeals quashed 39 convictions involving Horizon errors.  This scandal, which has been called the largest miscarriage of justice in the UK for many decades, will have legal repercussions for years.  But now that things are starting to be remedied, how did they get so bad in the first place?

I once knew a woman who had worked her way up to being postmaster of a small New England town.  She enjoyed her job until one day when about $20,000 of stamps turned up missing.  To this day I believe she was not guilty of stealing the stamps, but the U.S. Postal Service held her personally responsible for the loss, and when we left New England for Texas around the time Horizon was being installed in 2000, she was still paying off that debt. 

I’m not sure what it is about postal service managers that make them jump to the conclusion that any financial discrepancy is automatically the fault of the local person in charge, but that’s certainly what happened in the case of the Horizon system.  The 2015 investigation report, which was eventually obtained by news organisations, said that Horizon’s communications links were so bad that an average of 12,000 communication failures happened every year.  Horizon was developed by Fujitsu in the late 1990s mainly as a way to automate welfare benefit payments, which were then handled through the quasi-governmental Post Office branches.  The government’s Benefits Agency then pulled out, leaving Fujitsu to finish the job on its own. 

If one reads between the lines of the reports on this scandal, it seems that the errors happened like this:  A transaction involving cash takes place at a remote location, but there is a communications glitch between the remote station and the central accounting office.  Money goes out from the remote kiosk but doesn’t get reported to the main system.  Evidently, the system was not designed to do checks or other actions that would identify such dropouts and correct them.  When the physical cash was counted at the end of the reporting period, naturally it came up short.  

Despite the fact that the sub-postmaster in charge might know that the machine was giving out cash but not reporting it to headquarters, his bosses believed the machine, not him, and accused him of theft.

Multiply this scenario by a few dozen cases a year, and you have a real nightmare.  Fortunately, the nightmare is drawing to a close, but there is no way to recover the reputations and well-being of those who lost both when they were falsely accused of stealing. 

Synergism can be good or bad, and in this case it was terrible. You had a badly designed hardware and software system that was prone to errors, to begin with. And then it was installed by managers whose ignorance of technology led them to view computers as a sort of oracle of God: the machine can’t be wrong, so it must be those pesky humans that are stealing the money in devious ways we can’t detect.  And what is worse, once the management had taken that position, the longer time went on the harder it would be to admit they were wrong, and maybe all these prosecutions were a mistake after all. So, unwisely but understandably, the managers dug in their heels, even going to the extent of quashing the report that revealed they were wrong.

The people responsible for this fiasco may or may not pay a penalty for their cover-ups and denials.  Groups of present and former sub-postmasters are continuing to seek legal redress for their unjust convictions, and this may involve civil lawsuits that would penalize the managers who made bad decisions.

But regardless of what happens in the future, engineers everywhere can take this scandal as a bad example of how not to do an IT system. It is a remarkable thing that, at least in the U.S., there have been relatively few instances of internal failures in the money-machine networks run by banks, as opposed to attacks by outsiders. Commercial banks, being historically conservative institutions, apparently insisted from the outset on multiple checks and extreme robustness in their money-handling networks, so that even in the face of communications interruptions and power failures, they always know how much money they have and can keep track of it without loss.

Fortunately, the UK Post Office has announced that they are replacing Horizon with a cloud-based system that should work much better.  For the sake of its customers and especially for the well-being of its sub-postmasters, let’s hope they’re right.


Mind Matters News

Breaking and noteworthy news from the exciting world of natural and artificial intelligence at MindMatters.ai.

Who Will Your Boss Believe? You or That Glitzy New Computer?