195 incidents on record · 2026 Headlights Incident reports by Ellie Harris · Melbourne
11 new this week Library last updated 12 September 2026
← The incident library
HD-INC-178
Consumer AI · Canada · 2026 · Personal information collected without valid consent, and the general accuracy of personal information generated about people never assessed

Canadian privacy regulators found OpenAI had not validated the general accuracy of personal information generated about people

By Ellie Harris · Filed PIPEDA Findings #2026-002 published 6 May 2026; the investigation covered GPT-3.5 and GPT-4

Alleged: OpenAI developed or deployed the AI system implicated in this incident. Details are drawn from public reports; parties are presumed innocent of any wrongdoing not established by an official finding.

Canadian privacy regulators found OpenAI had not validated the general accuracy of personal information generated about people

What happened

It was reported in May 2026 that four Canadian privacy authorities published the findings of a joint investigation into OpenAI. The investigation covered GPT-3.5 and GPT-4, the models powering ChatGPT when the inquiry began. It did not assess later models or OpenAI’s other AI services.

It was reported that the regulators found OpenAI’s collection, use and disclosure of personal information from publicly accessible websites and licensed third party sources for training was too broad and inappropriate. It was reported that the federal, British Columbia and Alberta authorities also found OpenAI had not obtained valid consent for collecting and using personal information from publicly accessible sources to train the models.

The investigation also examined personal information generated by ChatGPT about people. It was reported that the federal, British Columbia and Alberta authorities found OpenAI had not met their accuracy requirements.

What an auditable version would have shown

It was reported that the regulators found OpenAI had carried out internal evaluations of model accuracy in areas such as mathematics and history, but had not assessed the general accuracy of personal information generated by the investigated models about individuals.

It was reported that they also found warnings about inaccurate answers were not prominent enough and did not specifically address the accuracy of personal information. GPT-3.5 did not provide sources, while GPT-4 provided them inconsistently when its browser search feature was triggered.

An auditable record could show what factual check ran when information about an identifiable person was generated, what source was used and what the check returned.

Where the gap was

It was reported that the regulators also examined what happened when someone wanted to access, correct or delete personal information about themselves. It was reported that this could become difficult where OpenAI could not verify the information related directly and uniquely to the person making the request.

A VerificationGate is designed to send a factual claim about an identifiable person to a current identified source for checking. A ConductRecord keeps the request, the check and what happened next. A ConstraintGate is designed to apply a rule before information enters a training pipeline.

These are Headlights designs. They are not remedies ordered by the Canadian regulators.

What governance should have looked like

It was reported that OpenAI disagreed with a number of the findings and described changes made during the investigation, including filtering and masking personal information in training data, new factuality evaluations, web search with cited sources, and changes to the handling of access and correction requests.

It was reported that the outcome differed across the four jurisdictions. The federal Privacy Commissioner found the seven issues examined well founded and conditionally resolved, Quebec reached different conclusions on some issues because its legislation differs, and British Columbia and Alberta found consent for the scraped data could not be obtained under their statutes.

Where a system generates factual information about identifiable people, best practice would be to test how accurately it does so before deployment, document the results and make the limitations clear to users. If someone asks to access or correct information about themselves, there should also be a workable record of what information was involved and what happened to their request.

Failure Pattern: it was reported that regulators found OpenAI had not assessed the general accuracy of personal information generated about individuals by the models they investigated.

Governance Principle: when an AI system generates factual information about identifiable people, the organisation operating it should be able to show how the accuracy of that information was tested and what happens when someone asks for it to be corrected.

The reference implementation of ConstraintGate, VerificationGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.

Sources

The mailing list

Fresh incident reports every week. One email to match.

We add new incidents to the library regularly, and send a single short email each week with what's new. The library stays free and open; this is just how you keep up with it.

No tracking. Unsubscribe in one click.

The record

An auditable system would have produced a signed, tamper-evident record the moment this happened: what the system did, the version that did it, the basis it acted on, and the action taken, and OpenAI could have produced it on demand.

This is the record the system as deployed did not produce in a signed, auditable form.

What this teaches
Capture what happened when it happens
What the system did, the version that did it, the basis it acted on, and the action taken, recorded at the moment, not reconstructed after.
Sign it, so no one has to trust the record-keeper
A tamper-evident entry. Edit it later and the signature breaks. The record does not ask for the benefit of the doubt.
Make it verifiable by anyone
A court, a regulator, a customer's lawyer can check the record themselves, without taking the company, or us, at our word.

Headlights summarises publicly reported AI incidents. All summaries are independently written, attributed to their original sources, and intended for research and educational purposes. Allegations are identified as such until established through official findings.

Last reviewed August 2026. This report is based on the sources listed above and reflects information available at the time of review; later developments may not be captured. Where a person is described as charged with or alleged to have done something, that allegation is unproven unless a conviction or a court or regulatory finding is stated. Headlights publishes journalism and commentary, not legal advice.

Want to write back?

Direct to my inbox.

ellie@useheadlights.com →