What happened
It was reported that in May 2026 the Pennsylvania Office of Attorney General announced an agreement with GEICO after a complaint from a customer whose new motor insurance policy had been cancelled.
It was reported that an AI tool selected the customer for an additional underwriting review. She was asked to provide documents and believed she had sent what was required. According to the state, she was not clearly told that what she had provided was not enough. Her policy was later cancelled and she was left driving without insurance.
The state alleged that the process was unfair and confusing, particularly because the final communication did not make clear that the documents she had provided would not prevent her policy from being cancelled.
Without admitting any violation, GEICO agreed to make several changes, including giving customers more time to provide documents, reducing some documentation requirements and training customer service staff on the changes.
What an auditable version would have shown
An auditable version should have shown the whole sequence, not just the AI decision at the beginning.
Why was this customer selected for additional review? What information did the AI use? What documents was she asked to provide? What did she send back? Was it enough? If not, when and how was she told?
It should also have been possible to see exactly when the policy was cancelled and whether the customer had been clearly warned before it happened.
Across all customers selected for review, those records could then show how many people were selected by the AI, how many later had their policies cancelled and how many had submitted documents before cancellation.
Where the gap was
The gap was not simply that AI was being used.
It was reported that the AI tool selected the customer for additional review. The problem alleged by the state happened in what followed.
The customer believed she had provided what was required, but according to the state, the communication she received did not make clear that her documents were not enough to prevent cancellation.
The gap was between the automated decision to select someone for review and making sure that person clearly understood what they needed to do next, what was still missing and what would happen if the requirement was not met.
A ConductRecord is designed to keep the selection, its inputs, the request sent to the customer, what came back and what was decided about it, so the sequence can be reconstructed from the record rather than from a complaint. A ConstraintGate is designed to test an action against a standing rule before it takes effect, here that a cancellation would not proceed until a recorded notice had told the person that what they sent does not meet the requirement.
The announced terms do not mention either design. ConductRecord and ConstraintGate are Headlights designs.
What governance should have looked like
It was reported that GEICO agreed to follow Pennsylvania’s guidance on the use of AI by insurers. The guidance makes clear that decisions or actions involving consumers still have to comply with insurance laws and regulations when AI or other advanced technology is involved.
Good governance should have followed the customer through the whole process.
If an AI system selects someone for a review that could eventually result in their insurance being cancelled, best practice would be to keep a clear record of why they were selected, what they were asked to provide, what they provided and what happened next.
Most importantly, before a policy is cancelled, the customer should be clearly told that they have not met the requirement and what will happen if they do not.
The AI decision might be the beginning of the process. Governance cannot stop there.
Failure Pattern: it was reported that an automated tool selected a new policyholder for further underwriting review, that the letter telling her what to send did not make clear that what she had sent was not enough, and that her cover was cancelled without her knowing. Governance Principle: where an automated step selects a person for a review that can end in them losing something, the organisation should be able to show why they were selected, what they were asked for, what they sent, and what they were told about whether it was enough. The reference implementation of ConductRecord and ConstraintGate is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Reporting carrying the terms of the agreement (Insurance Journal, 26 May 2026)
- Notice 2024-04, Use of Artificial Intelligence Systems by Insurers, Pennsylvania Bulletin, 6 April 2024, 54 Pa.B. 1910