What happened
It was reported that in November 2021 the Office of the Australian Information Commissioner found that the Australian Federal Police had breached its privacy obligations by using Clearview AI’s facial recognition tool without first conducting a required privacy impact assessment. The tool was trialled by the Australian Centre to Counter Child Exploitation, a unit the AFP leads, which took up free trial accounts offered by Clearview between November 2019 and January 2020. According to the determination, ten members registered and seven of them ran searches, uploading facial images of Australians, including images of persons of interest and victims in live investigations, to a service that had built its database by scraping billions of photos from the web.
According to the determination, Commissioner Angelene Falk found that the AFP had failed to carry out a privacy impact assessment before the trial, which the Australian Government Agencies Privacy Code requires for any project that carries high privacy risk, and that it had breached Australian Privacy Principle 1.2 by not having the practices, procedures and systems needed to ensure its use of the tool complied with the Privacy Act and the Australian Privacy Principles. There were, she said, “a number of red flags about this third party offering that should have prompted a careful privacy assessment,” and by uploading information about persons of interest and victims the unit was handling personal information “in a way that could have serious consequences for individuals.” She directed the AFP to bring in an independent assessor to review and fix the gaps in its privacy practices, and to make sure relevant staff completed updated privacy training. The AFP had argued the exercise was a limited trial that did not require an assessment; the Commissioner did not accept that. This finding was separate from the OAIC’s earlier determination against Clearview AI itself, which in October 2021 found the company had breached Australians’ privacy by scraping their images to build a facial recognition database.
What an auditable version would have shown
The problem was not that a tool was tested, but that a high-risk test started before the privacy assessment that was supposed to come first. An auditable version keeps that check on the record: a note, before the first image is uploaded, that a privacy impact assessment exists, who signed it off, and on what terms. Alongside it would sit a record of the use itself, which images were sent to the outside service, by whom, and for which investigation. With those two records, the question the regulator had to reconstruct afterwards, what personal information went where and whether anyone had weighed the risk, could have an answer on file before the trial started.
Where the gap was
A serious step, sending Australians’ facial images to an outside company, happened before the assessment that was meant to come first. A ConstraintGate turns that requirement into a check the project has to clear before it runs: with no completed high-risk assessment on file, the uploads do not go out. A ConductRecord keeps what was actually done, the images handled, the searches run, and the investigation each was tied to, so the agency can account for how it used people’s information from its own files rather than from a regulator’s inquiry. Neither decides whether facial recognition is a good idea. They just make sure the decision, and the reason for it, are on the record before the data leaves.
What governance should have looked like
When a public agency takes up a powerful new tool, especially one built on other people’s data, it is worth asking whether the use is lawful and proportionate before the trial begins, rather than leaving that question until a regulator raises it. A privacy assessment is the record that someone actually weighed the risk to the people whose faces were being uploaded. What this finding shows is that “it was only a trial” does not excuse skipping that step, and that an agency should be able to show, from its own files, that it checked before it acted and can say exactly what it did.
Failure Pattern: a government agency put a high-risk biometric tool to use without first assessing its privacy impact, and without having the practices, procedures and systems needed to keep that use compliant with privacy law.
Governance Principle: a system that handles people’s personal information at high risk should pass a documented risk assessment before it is used, and its use should be recorded so the handling of that information can be accounted for.
The reference implementation of ConstraintGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- AFP ordered to strengthen privacy governance (OAIC media release, 16 December 2021)
- Commissioner Initiated Investigation into the Australian Federal Police (Privacy) [2021] AICmr 74 (26 November 2021)
- Commissioner Initiated Investigation into Clearview AI, Inc. (Privacy) [2021] AICmr 54 (14 October 2021)