What happened
It was reported that the UK Home Office introduced the automated Streaming Tool in 2015. The Independent Chief Inspector of Borders and Immigration recorded that it used Global Visa Risk Status data, including nationality, to assign visa applications a red, amber or green risk level that determined the scrutiny an application received. The inspection also recorded a ministerial authorisation allowing additional scrutiny by nationality. Applicants were not told their grades. When campaigners sought the country classifications they received a list with the country names redacted. TechCrunch reported that the Home Office had acknowledged a secret list of suspect nationalities but refused to provide meaningful information about the algorithm.
It was reported that the Joint Council for the Welfare of Immigrants and Foxglove brought judicial review proceedings alleging unlawful race discrimination under the Equality Act 2010. They alleged a feedback loop in which a higher risk grade for a nationality produced more refusals, which then raised the risk associated with that nationality. The allegation was never tested because the case did not reach a hearing, and no court determined whether the feedback loop existed or the tool discriminated unlawfully. The Home Office agreed to redesign the tool without accepting the allegations. Internal guidance suspended it on 6 August 2020 and was published six days later. The inspection report recorded that the department withdrew the tool across all entry clearance operations in response to the proceedings. Its replacement, the Complexity Application Routing Solution for Visits, does not use machine learning and does not decide applications, according to the Home Office’s algorithmic transparency record. Decision-makers assess each visitor visa application individually under the Immigration Rules and Visitor Policy Guidance.
What an auditable version would have shown
The challenge centred on whether grading by nationality produced different patterns of scrutiny and refusal among the nationalities the system sorted. No answer entered the public record, although the question was measurable. A system operating for five years could retain the distribution of applications across red, amber and green, the refusal rate for each grade, differences by nationality and changes over time. It could also preserve, for each application, the grade assigned, the factors behind it and the caseworker’s response. An applicant whose case received more scrutiny could then understand and contest the basis. Instead campaigners tried to reconstruct the system from outside and received country classifications with the countries removed.
Where the gap was
It was reported that the Streaming Tool used nationality to help determine how much scrutiny an application received. Applicants did not see their grades and the Home Office withheld the graded country list. A MetricRecord converts individual decisions into continuing population-level figures, allowing refusal rates by nationality and grade to be monitored routinely and disparities to become internal governance signals rather than matters first raised in court. A ConductRecord preserves what the system did in each case and how the human decision-maker responded, allowing an applicant to understand and contest the grade. The records answer different questions, one about the system’s effects across a population and the other about its effect on one person. The tool operated from 2015 until litigation prompted its withdrawal in 2020.
What governance should have looked like
It was reported that the UK Home Office did not accept the challengers’ allegations, and it was reported that it never had to answer them at a hearing because it withdrew the tool first. Where the state uses an automated grade to determine the scrutiny a person receives, governance needs population-level measures showing how grades and outcomes differ among the groups being sorted, with review triggered by a material disparity, and an individual record of why a grade was assigned, how it shaped the process and who remained responsible for the decision. Legitimate limits on public disclosure do not remove the need for internal scrutiny or a route to challenge. The replacement routing solution does not use machine learning. Five years of grading ended with serious but untested allegations, a withdrawn system and no published measurement showing what the grading had done.
Failure Pattern: an automated risk grade that took nationality as an input decided how much scrutiny a person’s application received, while the person could not see their grade and the criteria were withheld.
Governance Principle: where an automated grade determines the scrutiny a person receives from the state, its performance across the groups it sorts should be measured and published, and the basis of each grade should be recorded and open to challenge.
The reference implementation of MetricRecord and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Complexity Application Routing Solution - Visits, algorithmic transparency record (Home Office)
- An inspection of visit visa operations, December 2022 to January 2023 (Independent Chief Inspector of Borders and Immigration)
- Home Office says it will abandon its racist visa algorithm, after we sued them (Foxglove)
- Home Office to scrap ‘racist algorithm’ for UK visa applicants (The Guardian)
- Government scraps immigration “streaming tool” before judicial review (UK Human Rights Blog)