What happened
In September 2024, the Texas Attorney General announced a settlement with Pieces Technologies, a company that develops generative AI products used in healthcare, including tools that help medical staff summarise and draft clinical notes.
The state alleged that the company marketed the accuracy of its products using the terms “critical hallucination rate” and “severe hallucination rate”, with figures of “<.001%” and “<1 per 100,000”. However, the state alleged that the company had not adequately explained how those figures were calculated.
The state’s allegation was that representations about its generative AI products “may have violated the DTPA because they were false, misleading, or deceptive”.
The company denied violating the law and maintained that it had accurately represented its hallucination rate.
Under the settlement, the company agreed to requirements around how it represents the accuracy and performance of its AI products, including greater transparency about how metrics are defined and calculated.
What an auditable version would have shown
It was reported that the company advertised very low hallucination rates for its generative AI products. The problem is that a number like “fewer than one per 100,000” means very little unless we know what was counted, how it was tested, over what period and who decided whether an output was wrong.
An auditable version should have made that clear. It should have shown how the hallucination rate was calculated, what data was used, what counted as a hallucination and how the results could be checked independently.
It was also reported that the settlement required the company, when making claims about metrics or benchmarks, to explain what those measurements mean and how they were calculated. Alternatively, the company could use an independent third-party auditor to assess and substantiate its claims.
The important distinction is that the settlement did not require the company to prove that its earlier figures were wrong or right. The requirements applied to how these types of claims would be supported and communicated going forward.
Where the gap was
The gap was between the accuracy figures being reported and the evidence explaining how those figures were calculated.
It was reported that the company used very low hallucination rates when marketing its generative AI products, but the meaning behind those rates and the method used to calculate them were not clearly explained.
The settlement addressed this by requiring future claims about metrics or benchmarks to include what the measurement means and how it was calculated, or to be supported by an independent third-party auditor.
The gap was not simply whether the AI produced hallucinations. It was whether a customer could understand, verify and independently assess the accuracy claims being made about the system.
A MetricRecord is designed to compute an aggregate over signed records of what a system actually produced, with the chain of those records bound to the figure, so a rate carries its own denominator and period. A VerificationGate is designed to check a proposed output against a trusted source rather than against the model that produced it, here the drafted note against the encounter it summarises.
The settlement requires neither. MetricRecord and VerificationGate are Headlights designs.
What governance should have looked like
Good governance would have meant being clear about how the accuracy claims were reached in the first place.
It was reported that the settlement required the company to be transparent about its accuracy claims, testing methods, what its metrics mean and the data used to train its products. Customers must also be given information about intended use, known limitations and risks, and how to monitor for inaccurate outputs.
If a vendor gives a clinical customer an accuracy figure, best practice would be for the method behind it to come with the number. What counts as an error? What was tested? Over what period? And who decided whether an output was right or wrong?
A clinical buyer should be able to ask where an accuracy figure came from and get a clear answer. Without that, it is very difficult to know whether the number is meaningful.
Failure Pattern: it was reported that a vendor advertised hallucination rate figures to clinical buyers, that the state alleged the representations may have violated the deceptive trade practices act because they were false, misleading or deceptive, and that the remedy it took was to require future metric claims to carry their meaning and their method of calculation, or an independent auditor’s substantiation. Governance Principle: a published accuracy figure should be traceable to the outputs it was computed over, with its definition, its denominator and its period stated, so that a buyer can check the number rather than take it on trust. The reference implementation of MetricRecord and VerificationGate is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.