What happened
It was reported that when exams moved online during the pandemic, universities and exam boards turned to remote-proctoring software from companies such as Proctorio and ExamSoft. The software watched a student through their webcam while they sat an exam and used AI to detect the student’s face, follow their eyes and movements, and flag anything it judged to be a sign of cheating, such as looking away or another face appearing. A flag could then be reviewed, but the student was being judged, throughout, by a system reading them through a camera.
It was reported that the face-detection worked poorly on darker-skinned students. An analysis of the face-detection model used by Proctorio found that it failed to detect Black faces more than half the time, and a peer-reviewed study found disparities in how the software performed across skin tone, race and sex. Students described having to light their faces harshly or being unable to get the software to recognise them at all, which could stop them from starting the exam. In 2020 a group of US senators wrote to the proctoring companies raising concerns about bias and privacy. A student the software flagged could not see what had triggered it, and the companies defended their tools.
What an auditable version would have shown
Software that decides a student’s face cannot be seen, or that their behaviour looks like cheating, is making judgements the student cannot see and cannot answer. An auditable version keeps a record for each student showing what the system detected, what it flagged and why, and makes it available, so a student wrongly flagged, or wrongly told their face cannot be found, has something concrete to point to. It would also count how often the software fails to find a face or raises a flag, broken down by skin tone, so a tool that works worse for darker-skinned students is something the school and the vendor can see, not a pattern students have to prove one complaint at a time.
Where the gap was
Students were judged by software reading them through a camera, and its face-detection failed on darker-skinned students while its flags could not be seen or contested. A MetricRecord counts how often the system fails to find a face or raises a flag across skin tones, so a tool that performs worse for some students is measured before it is relied on, not discovered by the students it fails. A ConductRecord keeps each flag and what triggered it, and puts it in the student’s hands, so a wrong flag can be shown and put right. The disparity here was found by an outside analysis of the model. A record kept inside the software would have shown the same thing from the start.
What governance should have looked like
Where software judges people through a camera, it should be tested and shown to work across skin tones before anyone is made to rely on it, and each judgement it makes should be recorded and open to the person to see and challenge. Best practice would be for the vendor and the school to measure and publish how often the system fails to detect a face or raises a flag across different groups of students, and to keep, for each student, what was flagged and why, available to them. The companies said their tools worked. Whether they worked for a darker-skinned student trying to start an exam is something a measurement across skin tones would show.
Failure Pattern: exam-surveillance software judged students through a webcam and its face-detection failed on darker-skinned students, locking some out or flagging them, while judging behaviour the student could not see or contest.
Governance Principle: where software judges people through a camera, its detection must be tested and shown to work across skin tones before it is relied on, and each flag and its basis must be recorded and open to the person to see and challenge.
The reference implementation of MetricRecord and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Proctorio is using racist algorithms to detect faces (Vice / Motherboard)
- Racial, skin tone, and sex disparities in automated proctoring software (Frontiers in Education, 2022)
- Senator: more transparency is needed by exam proctoring tech firms (TechCrunch)