What happened
It was reported that France, for the 2024 Paris Olympic and Paralympic Games, became the first country in the European Union to write mass algorithmic video surveillance into law. A 2023 law let the police and private contractors run software over the feeds from cameras and drones to watch for a set of events in real time: a sudden change in the size or movement of a crowd, an abandoned bag, a weapon, a person on the ground, fire or smoke, and certain traffic offences. The software was used before, during and after the Games, and the companies behind it included Videtics, Wintics, ChapsVision and Orange Business.
It was reported that the French government said the system did not identify anyone, that it looked for events rather than faces, and that it was built to avoid the use of biometric data that could pick a person out. Civil liberties groups and legal scholars disputed this, arguing that software which reads people’s bodies, their posture, their gait and their movements is analysing biometric data whether or not it puts a name to them, and that a wide-reaching surveillance power was being normalised through a sporting event and set to outlast it. The law framed the system as an experiment, authorised to run for a period around the Games, and there were moves to keep it in place afterwards.
What an auditable version would have shown
A system watching a crowd of millions makes a great many judgements, and from the outside hardly any of it is visible: how often it flagged something, what it flagged, whether the flag was right, and whether it fell more heavily on some people or places than others. An auditable version keeps a signed record of each alert, what triggered it and what followed, and keeps the aggregate, how often the system fires and how often it is wrong, so the public and an oversight body can judge whether it works and whether it is fair. The claim that the system never identifies anyone is also something a record can hold to account: a log of the data each alert rested on shows whether the line into biometric identification was crossed, rather than leaving it to be asserted.
Where the gap was
A powerful surveillance system was run over the public on the strength of the state’s assurance about what it did and did not do, with nothing published that anyone outside could use to test the assurance. A ConductRecord keeps each alert, what triggered it and what data it used, so whether the system stayed within its limits, and in particular whether it stepped into identifying people, can be checked rather than trusted. A MetricRecord keeps how often it fires and how often it is wrong, across places and groups, so a system that flags some people more than others shows up as a figure. The safeguard the law leaned on was a promise not to identify anyone. Whether that promise held is the kind of thing a record can show.
What governance should have looked like
Where the state runs an automated system over the public, the case for it cannot rest on the state’s own account of what it does. There has to be a record, open to an independent body, of what the system flagged, on what basis, and how often it was wrong, and a hard limit, provable from that record, on what it is not allowed to do, such as identifying individuals. Best practice would be for the operator to log every alert and the data behind it, to publish how often the system fires and errs, and to make the no-identification rule a check that runs and is recorded, not just a line in a statute. France put the power into law. What it did not put in place was the record that would let anyone outside the system see whether the limits were being kept.
Failure Pattern: the state ran an automated surveillance system over the public and asked to be trusted on what it did and did not do, with nothing published that would let anyone outside see what the system flagged, how often it erred, or whether it stayed within its stated limits.
Governance Principle: where the state runs an automated system over the public, an independent record must show what it flagged, on what basis and how often it was wrong, and any hard limit, such as not identifying individuals, must be enforced and provable from that record rather than merely asserted.
The reference implementation of ConductRecord and MetricRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Paris 2024 Olympics: Concern over French plan for AI surveillance (BBC)
- AI mass surveillance at Paris Olympics: a legal scholar on the security boon and privacy nightmare (The Conversation)
- Paris Olympics will be a training ground for AI-powered mass surveillance (Scientific American)