180 incidents on record · 2026 Headlights Incident reports by Ellie Harris · Melbourne
10 new this week Library last updated 30 August 2026
← The incident library
HD-INC-157
Government · United States · 2025 · Unverified AI-assisted contract triage

ProPublica reported that the tool used to pick Veterans Affairs contracts for cancellation was shown only the first 10,000 characters of each contract, and recorded around 1,100 of them as worth 34 million dollars each

By Ellie Harris · Filed Contract review conducted from February 2025

Alleged: US Department of Veterans Affairs; Department of Government Efficiency developed or deployed the AI system implicated in this incident. Details are drawn from public reports; parties are presumed innocent of any wrongdoing not established by an official finding.

ProPublica reported that the tool used to pick Veterans Affairs contracts for cancellation was shown only the first 10,000 characters of each contract, and recorded around 1,100 of them as worth 34 million dollars each

What happened

It was reported that an executive order signed in February 2025 gave federal agencies 30 days to complete a review of their contracts and grants, and that at the Department of Veterans Affairs the job of working out which contracts to cancel went to a tool built for the purpose. ProPublica reported on 6 June 2025 that the tool was written by a software engineer working with the Department of Government Efficiency, who had close to 15 years in software engineering and no formal training in artificial intelligence. ProPublica obtained the source code and the prompts and published them. It reported that the tool ran on older and inexpensive OpenAI models already available through the department, and that it was shown the first 10,000 characters of each contract, about 2,500 words, which ProPublica described as the opening portion of the contract, containing limited summary information. Before judging a contract the tool was told to pull out a set of fields: the contract number, the vendor, the contracting officer and the total contract value. The prompt then told it to mark for cancellation anything that did not directly support patient care, protecting medical equipment maintenance, clinical procedures and compliance audits, and targeting diversity initiatives, administrative support and work it judged could be brought in house. Contracts it judged cancellable were labelled with a word of its author’s coinage, munchable.

It was reported that the tool flagged more than 2,000 contracts, and that its output carried values the contracts did not support. ProPublica reported that it recorded around 1,100 agreements as worth 34 million dollars each when some were worth as little as 35,000 dollars, and put the errors down to the model being handed an outdated version of a contract, or lifting an irrelevant number off the page and treating it as the contract value. Internet service was marked for cancellation. So was a maintenance contract for ceiling lifts, equipment the contract itself identified as safety critical and used to reposition patients during their care. The engineer told ProPublica that mistakes had been made, that he would not recommend anyone run his code and do what it says, and that no contracts were terminated without human review. The VA press secretary said decisions to cancel or reduce contracts were made after multiple reviews by department staff including contracting experts and senior officials, and VA officials said nearly 600 contracts had been killed overall. ProPublica reported that the relationship between the tool’s flags and individual cancellations was not clear. On 24 June 2025 two US senators wrote to the VA Acting Inspector General asking for an investigation, setting out preliminary findings that included more than 650 contracts terminated for convenience between 20 January and 30 May 2025. The senators wrote that officials had used artificial intelligence formulas and algorithms to make or inform cancellation decisions, which is their characterisation and not a finding by the Inspector General.

What an auditable version would have shown

The value of a federal contract is not a matter of opinion. It sits in the contract, and in the systems that administer it. A model can help find it, and the decision process still has to resolve it against the contracting record rather than take what the model returned. An auditable version does exactly that, and refuses to move on a figure that does not match. For each contract it keeps what text the model was actually shown, which model version answered, what came back, and which named official then decided. That record answers the questions the reporting can only leave open: how many of the nearly 600 cancellations traced back to a flag, how many of those flags carried a value the contract did not support, and how many a reviewer changed before the cancellation went out.

Where the gap was

It was reported that the tool recorded around 1,100 agreements as worth 34 million dollars each, including agreements worth as little as 35,000 dollars, and that the review ran against a 30 day deadline. A VerificationGate sends a factual claim to the system that holds the fact rather than back to the model that proposed it, so a contract value comes from the contract register and a mismatch stops the record instead of travelling with it. A ConductRecord keeps the input, the model version, the output and the human response for each contract, which is what lets anyone establish afterwards whether a cancellation rested on a figure the model got wrong. The reporting describes a model-generated list of flagged contracts informing the review that followed, and it does not show which flags led to which cancellations. That is the traceability an auditable process exists to provide.

What governance should have looked like

It was reported that the review was completed under a 30 day deadline, and that the code and the prompts became public because the engineer published them rather than through any departmental disclosure. Where automated triage feeds decisions that end contracts, good practice is for facts about each record to be read from the system of record rather than generated, for each flag to be traceable to the text it rested on, and for the decision that follows to name the person who made it. The engineer said no contract was terminated without human review, and the VA press secretary said decisions followed multiple reviews. Nothing has been published that would let anyone outside the department match a cancellation to the flag that preceded it. No VA Office of Inspector General report responding to the June 2025 request was identified in the sources reviewed.

Failure Pattern: a language model was asked for facts about documents it had only partly been shown, and the figures it returned entered a decision process without being resolved against the records that hold them.

Governance Principle: where a model makes a claim about a named record, its value, status, obligation, owner, risk or eligibility, that claim should be resolved against the authoritative system of record before it informs a decision, and the file should show which person decided what followed.

The reference implementation of VerificationGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.

Sources

The mailing list

Fresh incident reports every week. One email to match.

We add new incidents to the library regularly, and send a single short email each week with what's new. The library stays free and open; this is just how you keep up with it.

No tracking. Unsubscribe in one click.

The record

An auditable system would have produced a signed, tamper-evident record the moment this happened: what the system did, the version that did it, the basis it acted on, and the action taken, and US Department of Veterans Affairs; Department of Government Efficiency could have produced it on demand.

This is the record the system as deployed did not produce in a signed, auditable form.

What this teaches
Capture what happened when it happens
What the system did, the version that did it, the basis it acted on, and the action taken, recorded at the moment, not reconstructed after.
Sign it, so no one has to trust the record-keeper
A tamper-evident entry. Edit it later and the signature breaks. The record does not ask for the benefit of the doubt.
Make it verifiable by anyone
A court, a regulator, a customer's lawyer can check the record themselves, without taking the company, or us, at our word.

Headlights summarises publicly reported AI incidents. All summaries are independently written, attributed to their original sources, and intended for research and educational purposes. Allegations are identified as such until established through official findings.

This report is based on the sources listed above and reflects information available at the time of review; later developments may not be captured. Where a person is described as charged with or alleged to have done something, that allegation is unproven unless a conviction or a court or regulatory finding is stated. Headlights publishes journalism and commentary, not legal advice.

Want to write back?

Direct to my inbox.

ellie@useheadlights.com →