180 incidents on record · 2026 Headlights Incident reports by Ellie Harris · Melbourne
10 new this week Library last updated 30 August 2026
← The incident library
HD-INC-171
Government · South Africa · 2025 · Automated eligibility screening that could not distinguish income from any other deposit

South Africa's social grant agency read every deposit into an applicant's bank account as income, and the High Court found its checks against government databases unreasonable and unfair

By Ellie Harris · Filed Grant introduced May 2020; regulations challenged in case 071891/2023; judgment 23 January 2025

Alleged: South African Social Security Agency developed or deployed the AI system implicated in this incident. Details are drawn from public reports; parties are presumed innocent of any wrongdoing not established by an official finding.

South Africa's social grant agency read every deposit into an applicant's bank account as income, and the High Court found its checks against government databases unreasonable and unfair

What happened

It was reported that the High Court in Pretoria found in January 2025 that parts of the system used to decide eligibility for South Africa’s monthly Social Relief of Distress grant were unconstitutional. Applications could be made online only. Eligibility was then tested using the applicant’s bank account information and matching the applicant against government databases. The court declared the income threshold of R624 a person a month invalid, and declared the online only application channel invalid as well.

The judgment records that the bank check considered only money deposited into the account and did not consider the purpose for which it was deposited. It also records that the parties accepted the government databases as unreliable because they were not updated regularly. The court found the database check unreasonable and unfair and used with an ulterior purpose of excluding eligible applicants. The state appealed, and the Supreme Court of Appeal heard the appeal on 25 August 2026 and reserved judgment.

What an auditable version would have shown

The judgment records that the process tested a deposit without testing what the deposit was for. It also records that the databases it matched against were not being updated regularly. Applicants who were refused received the outcome. It was reported that counsel for the applicants told the Supreme Court of Appeal in August 2026 that some sources indicate database exclusion errors may be as high as 33 per cent. Counsel also said there are around one million appeals a month and 12 million a year, of which 99 per cent are refused. Those figures were put to a court by a party to the case. The sources reviewed do not show the agency publishing figures of its own.

For each decision, the agency could keep a record of the check it ran, the data it used, how old that data was and what the check found. Those records could then show how many people were refused because of a single deposit where the purpose of the money was unknown, and how many were refused using database information that was older than their application.

Where the gap was

The judgment records that the bank verification process did not permit any consideration of the purpose for which money was deposited in an applicant’s account. A ConstraintGate is designed to test an action against the declared standing rule before it takes effect. Where the available data cannot satisfy that rule, it is designed to require an explicit recorded approval. A VerificationGate is designed to route a factual question, such as whether a person has income, to a trusted source rather than to an inference drawn from a bank balance. A ConductRecord preserves the inputs, the checks and the outcome. That allows a refusal to be explained afterwards rather than reconstructed. None of these decides who should qualify for a grant, and none of them settles the questions now before the Supreme Court of Appeal.

What governance should have looked like

The judgment records that the applicants challenged the agency’s explanation to Parliament in September 2023 that people had not been paid because they had not supplied their new banking details, arguing that the explanation was not plausible. It was reported that counsel for the department told the Supreme Court of Appeal that the online system is what allows it to reach 15 million people a month, and that kiosks now exist at agency offices for people without electronic access. It was also reported that counsel for the applicants replied that kiosks had been piloted at some offices only. The appeal is unresolved.

Where a public body uses an automated check to decide whether someone gets a benefit, best practice would be to make sure the check is actually answering the right question before using it to refuse someone, and to tell the person why they were refused when the decision is made.

For each decision, the agency could keep a record showing what check was run, what data it used, how old that data was and the result. Those records could show how many people were refused because of a single deposit where it was unclear what the money was for, and how many were refused based on database information that was older than their application.

Failure Pattern: it was reported that the automated process treated money deposited into a bank account as income without checking what the money was for. It also relied on government databases that were not regularly updated, leaving people who were refused unable to see which check or information led to the decision.

Governance Principle: when a public body uses an automated check to refuse someone a benefit, it should be able to show what was checked, what information was used, how current it was and what the check found.

The reference implementation of ConstraintGate, VerificationGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.

Sources

The mailing list

Fresh incident reports every week. One email to match.

We add new incidents to the library regularly, and send a single short email each week with what's new. The library stays free and open; this is just how you keep up with it.

No tracking. Unsubscribe in one click.

The record

An auditable system would have produced a signed, tamper-evident record the moment this happened: what the system did, the version that did it, the basis it acted on, and the action taken, and South African Social Security Agency could have produced it on demand.

This is the record the system as deployed did not produce in a signed, auditable form.

What this teaches
Capture what happened when it happens
What the system did, the version that did it, the basis it acted on, and the action taken, recorded at the moment, not reconstructed after.
Sign it, so no one has to trust the record-keeper
A tamper-evident entry. Edit it later and the signature breaks. The record does not ask for the benefit of the doubt.
Make it verifiable by anyone
A court, a regulator, a customer's lawyer can check the record themselves, without taking the company, or us, at our word.

Headlights summarises publicly reported AI incidents. All summaries are independently written, attributed to their original sources, and intended for research and educational purposes. Allegations are identified as such until established through official findings.

Last reviewed August 2026. This report is based on the sources listed above and reflects information available at the time of review; later developments may not be captured. Where a person is described as charged with or alleged to have done something, that allegation is unproven unless a conviction or a court or regulatory finding is stated. Headlights publishes journalism and commentary, not legal advice.

Want to write back?

Direct to my inbox.

ellie@useheadlights.com →