Horizon: the law presumed the computer was right

TL;DR. Between 1999 and 2015 the UK Post Office prosecuted hundreds of sub-postmasters for theft and false accounting on the evidence of Horizon, its branch accounting system. In December 2019 the High Court found Horizon had not been sufficiently robust and that the Post Office had shown a pattern of defensiveness and a lack of transparency. The Court of Appeal quashed 39 convictions in 2021 as an abuse of process, and Parliament quashed the rest on a blanket basis in 2024. There is no artificial intelligence anywhere in this case. It is a transaction-processing system with bugs. It belongs in this record because the mechanism that turned software defects into wrongful convictions was legal, not technical: English law presumed computer evidence reliable unless the accused could show otherwise, and the accused had no access to the system.

---

Status: established. Primary sources: Bates & Others v Post Office Ltd (No 6, "Horizon Issues") [2019] EWHC 3408 (QB), Mr Justice Fraser, 16 December 2019; Hamilton and others v Post Office Ltd [2021] EWCA Crim 577; and the Post Office (Horizon System) Offences Act 2024. Findings and quoted characterisations are the court's.

---

Horizon was installed across Post Office branches from 1999 to handle branch accounting. When a branch's accounts showed a shortfall, the sub-postmaster was contractually liable for it.

Shortfalls appeared. Sub-postmasters reported them, were told they were the only one experiencing the problem, and were required to make good the difference from their own money. Some remortgaged. Some went bankrupt.

And hundreds were prosecuted, many by the Post Office itself, which held the power to bring private prosecutions. Convictions for theft, fraud and false accounting followed. People went to prison. Marriages ended. At least some of those wrongly convicted died before their names were cleared.

The system had bugs. That fact was established in court twenty years after the deployment began.

What the court found

The civil group action was brought by 555 claimants under a Group Litigation Order made in March 2017 and heard by Mr Justice Fraser, who delivered six judgments. The one on the software is Judgment No. 6, the "Horizon Issues" judgment, on 16 December 2019.

The court found Horizon was not sufficiently robust and had suffered from bugs, errors and defects capable of producing discrepancies in branch accounts.

It found the Post Office had shown a pattern of defensiveness and a lack of transparency.

And Fraser J invited the Director of Public Prosecutions to investigate the conduct of Fujitsu, the system's supplier, in relation to evidence given in prosecutions.

In 2021 the Court of Appeal quashed 39 convictions, holding the prosecutions to have been an abuse of process. Three convictions were upheld on the basis that Horizon evidence had not been essential to those particular cases.

In 2024 Parliament passed the Post Office (Horizon System) Offences Act, which quashed the remaining convictions on a blanket basis rather than requiring each person to appeal individually. That is an extraordinary legislative act, and the reason for it is that the ordinary appellate route was too slow and too demanding for the number of people wrongly convicted.

A statutory public inquiry, upgraded from a non-statutory one in June 2021, continues.

Why this belongs in a record about AI

It does not contain any. Horizon is conventional transaction-processing software. No model, no training data, no probabilistic output. Everything in it was written by people as explicit instructions.

And that is the argument. Every failure mode this record has documented so far appears here, in a system with none of the properties usually blamed.

An output treated as more reliable than it was. In Moffatt it was a chatbot's answer. Here it was a balance figure.

A process built for a different kind of evidence. In Williams a similarity ranking entered a photo lineup. Here a software-generated discrepancy entered a criminal prosecution.

Institutional certainty overriding contrary reports. In the Dutch benefits scandal it was a culture that treated the flagged as guilty. Here it was hundreds of sub-postmasters, independently, in different parts of the country, being told each was the only one.

None of those required machine learning. If the mechanism does not need AI, then fixing AI does not fix the mechanism.

The presumption is the finding

Here is the specific legal feature that converted software defects into criminal convictions, and it is the most transferable thing in the case.

English law presumed that a computer system was operating correctly unless the party challenging it produced evidence to the contrary.

Consider what that requires of a defendant. To rebut the presumption, a sub-postmaster accused of theft would need evidence that the system had malfunctioned. The evidence of malfunction sits inside the system, which is owned by the prosecutor, and which the defendant cannot access. The bug reports, the error logs, the record of remote access to branch accounts: all held by the party bringing the case.

The burden was placed on the person least able to discharge it, in favour of the party best able to conceal it.

That is not a technology problem. It is an evidential rule that made a specific class of institution effectively unfalsifiable, and it applies with more force, not less, to systems whose outputs are probabilistic and whose logic is proprietary.

Four things this establishes

Reliability is a claim requiring evidence, not a default. Horizon was treated as correct because it was a computer. Nothing tested that until litigation forced it, twenty years in.

Independent reports of the same anomaly are data. Hundreds of people in different branches reported the same class of problem, and each was told they were alone. That pattern was itself the strongest available evidence, and the institution holding it was the one denying it existed.

Prosecutorial power plus system ownership is a structural conflict. The Post Office brought its own prosecutions using evidence from its own system, and controlled disclosure of the material that would have undermined it.

And a remedy can be too slow to be a remedy. Individual appeal was the correct legal route and it was so slow that Parliament eventually legislated around it. Where an automated process produces wrongful outcomes at scale, case-by-case correction is not a proportionate answer.

What it does not establish

That the software was the cause of every shortfall. The court found the system capable of producing discrepancies. It did not find that every disputed shortfall was a bug, and some prosecutions may have concerned actual dishonesty. Three convictions were upheld for that reason.

That AI systems face the same evidential presumption today. The presumption applies to computer evidence generally, and the position has been under active review since. Whether and how it now applies to probabilistic systems is unsettled.

The total count. The number prosecuted is commonly given in the hundreds and is still being established by the inquiry. Blanket quashing removed the need for individual determinations, which means the precise figure may never be fixed.

And who knew what, when. That is the inquiry's central question and its final report has not yet been published.

What is unresolved

Whether the presumption changes. The obvious reform is to require a party relying on computer evidence to demonstrate reliability rather than assume it. That would place the burden on the party with access, and it has not yet been enacted.

Whether compensation completes. Four redress schemes exist, and their slowness has been criticised repeatedly, including by people the schemes were built for.

Whether anyone is held individually accountable. The DPP was invited to investigate in 2019.

And whether the lesson generalises in practice. Automated decision systems are now used across benefits, tax, immigration and policing. The presumption in favour of the machine has not obviously weakened in any of them.

The counter-argument

This is not an AI incident and including it dilutes the record. Horizon is deterministic software with defects. An incident record for AI that admits any computer failure loses the boundary that makes it a record of anything. The mechanism may generalise, but so does the mechanism behind most institutional failure, and that is not usually taken as grounds for inclusion.

The failure was institutional, and the software is nearly incidental. A Post Office that investigated the first fifty identical reports would have found the problem regardless of what produced the discrepancies. Framing this around the system risks excusing the people who decided to prosecute.

The presumption was reasonable when written. Requiring every prosecution to prove the reliability of every till, calculator and ledger system from first principles would make routine cases unmanageable. The rule exists for good reasons and the failure was in its application to a system nobody had tested at all.

And the remedy may set a difficult precedent. Parliament quashing convictions on a blanket basis, without individual determination, is a serious constitutional step. It was probably right here, and it is not obviously a template.

The short version

From 1999 the UK Post Office ran Horizon for branch accounting. Shortfalls appeared, sub-postmasters were held contractually liable, and hundreds were prosecuted, many by the Post Office itself using its private prosecution power. People were imprisoned, bankrupted, and in some cases died before being cleared.

In December 2019 the High Court found in Bates & Others that Horizon had not been sufficiently robust, that it suffered bugs and errors capable of producing the discrepancies, and that the Post Office had shown a pattern of defensiveness and a lack of transparency. The Court of Appeal quashed 39 convictions in 2021 as an abuse of process. Parliament quashed the remainder on a blanket basis in 2024, because individual appeal was too slow for the number of people involved.

There is no artificial intelligence in this case at all. It is conventional software with defects.

It belongs here because the mechanism was legal rather than technical. English law presumed a computer system correct unless the challenging party showed otherwise, and the evidence of malfunction sat inside a system owned by the prosecutor. The burden fell on the person least able to discharge it, in favour of the party best able to conceal it.

That presumption applies with more force to probabilistic, proprietary systems than to a deterministic one. A sub-postmaster at least knew what a correct balance would look like. Someone denied a benefit by a model cannot state what the right output was, let alone prove the system did not produce it.

And the pattern across this record now has a case with none of the technology in it. An output trusted beyond its reliability, entering a process built for different evidence, with institutional certainty overriding hundreds of independent contrary reports. If the mechanism does not need AI, fixing AI does not fix the mechanism.

Common questions

What was the Post Office Horizon scandal? From 1999 the UK Post Office deployed Horizon, a branch accounting system. It produced shortfalls in branch accounts for which sub-postmasters were contractually liable. Many were required to repay money they had not taken, and hundreds were prosecuted for theft, fraud or false accounting, often by the Post Office itself using its power to bring private prosecutions. The system was later found by the High Court to have contained bugs and errors capable of causing those discrepancies.

What did the High Court actually find? In Bates & Others v Post Office Ltd, Judgment No. 6 on the Horizon Issues, handed down on 16 December 2019, Mr Justice Fraser found that Horizon had not been sufficiently robust and had suffered from bugs, errors and defects. He found the Post Office had shown a pattern of defensiveness and a lack of transparency, and invited the Director of Public Prosecutions to investigate the conduct of the system's supplier in relation to evidence given in prosecutions. The civil action had 555 claimants.

Why is a case with no AI in an AI incident record? Because every failure mode documented in this record appears in it, in a system with none of the properties usually blamed. An output trusted beyond its demonstrated reliability, entering a process designed for a different kind of evidence, with institutional certainty overriding independent contrary reports. None of that requires machine learning. If the mechanism does not need AI, then fixing AI does not fix the mechanism, and a record that only examines AI failures will keep mistaking the technology for the cause.

What was the legal presumption and why does it matter? English law presumed that a computer system was operating correctly unless the party challenging it produced evidence to the contrary. To rebut it, a defendant needed evidence of malfunction, and that evidence sat inside a system owned by the prosecuting party and inaccessible to the accused. The burden fell on the person least able to discharge it, in favour of the party best able to conceal it. That presumption applies with greater force to probabilistic and proprietary systems, where a person cannot even state what the correct output would have been.

How were the convictions overturned? In 2021 the Court of Appeal quashed 39 convictions in Hamilton and others v Post Office Ltd, holding the prosecutions to be an abuse of process. Three convictions were upheld, on the basis that Horizon evidence had not been essential in those particular cases. In 2024 Parliament passed the Post Office (Horizon System) Offences Act, which quashed the remaining convictions on a blanket basis, because requiring each person to appeal individually was too slow for the number affected.

How many people were affected? The number prosecuted is commonly given in the hundreds and is still being established by the statutory inquiry. Because the 2024 Act quashed convictions on a blanket basis rather than individually, the precise figure may never be fixed. The civil group action had 555 claimants, which is a separate and smaller number than those prosecuted.

Has the law changed? Not yet in the relevant respect. The obvious reform is to require a party relying on computer evidence to demonstrate that the system was reliable, rather than presuming it and placing the burden of rebuttal on someone without access. That change has been widely argued for since the judgments and has not been enacted.

What should someone deploying an automated decision system take from this? That reliability is a claim requiring evidence rather than a default, that repeated independent reports of the same anomaly are the strongest signal available and are usually held by the party with the least interest in acting on them, that owning both the system and the process that acts on its output is a structural conflict, and that case-by-case correction is not a proportionate remedy when an automated process produces wrong outcomes at scale.