What happened
In 2020 the pandemic cancelled the summer exams in England. Rather than let schools’ estimated grades stand, the regulator Ofqual built a statistical model to standardise them, so that the national spread of A-level grades would look much like previous years. Teachers supplied, for each student, an estimated grade and a rank order within the class. The model then adjusted those using mainly the school’s historical distribution of grades in that subject and the prior attainment of the current cohort.
The results were issued on 13 August 2020. The model had left the teacher’s grade unchanged for about 59 percent of entries and raised it for around 2 percent. For the rest it cut. About 39 percent of A-level entries came out below the grade the teacher had given, most by a single grade and some by two or more. Because a grade now depended so much on a school’s past, a strong student in a school that had historically done poorly could be marked down for a record that was not theirs.
The cut did not fall evenly. Where a subject class at a school had fewer than five students, the model was not used at all and the teacher’s grade stood; for classes of five to fifteen a partial blend was applied; only larger classes were fully standardised. Small classes are more common in private schools, so the exemption tended to protect them, and independent schools saw the largest rise in top grades, up about 4.7 percentage points on the previous year, while sixth-form and further-education colleges rose only about 0.3. Ofqual’s own analysis found that a student’s individual background explained very little of who was downgraded, while which school they attended explained a great deal: the unfairness ran through the school, and the students it disadvantaged were disproportionately those in large state-school classes.
The response was immediate and national. Within days, after protests and the threat of legal action, the government reversed course. On 17 August 2020 it abandoned the calculated grades and awarded each student the higher of the algorithm’s grade or the teacher’s. Ofqual’s chief regulator stepped down later that month, and the Prime Minister, Boris Johnson, blamed a “mutant algorithm.” The Office for Statistics Regulation reviewed how the model had been built, and the Royal Statistical Society said it had been offered a role only under a years-long non-disclosure agreement and had declined, a characterisation Ofqual disputed.
What an auditable version would have shown
The questions after results day were ones Ofqual could answer only in aggregate: how many were downgraded, and how the cut fell across kinds of school. What no student could get was the specific basis for their own grade, and no independent statistician could examine the model without signing away the right to comment on it. An auditable version records each grade as a signed decision with the inputs behind it, the school’s history, the cohort’s prior attainment, the rank, and the taper applied, and computes a standing measure of how outcomes differ across school types and class sizes before results are released. With that, the 39 percent cut and the way the small-class exemption favoured private schools are numbers visible inside the system in advance, open to scrutiny, rather than a pattern the country discovered the morning the grades arrived.
Where the gap was
Three things were missing. A student’s grade was allowed to be decided largely by their school’s past, an attribute the student could not change and had not earned. Nothing measured how the model’s outcomes broke down by school type and cohort size before it went live. And there was no per-student, contestable record of how any grade had been reached, while independent statisticians said the terms offered for outside scrutiny were too restrictive to accept. A ConstraintGate is the control on the first: a decision about an individual is not allowed to be driven mainly by a group attribute they cannot change, such as their school’s history. A MetricRecord is the control on the second: a signed, recomputable measure of outcomes across groups, so disparate impact is seen before release rather than after protest. A ConductRecord is the control on the third: a record of each grade and its inputs that a student can see and challenge, and that an independent reviewer can verify openly.
What governance should have looked like
A school-leaving grade decides university places and futures, which makes it as consequential as any automated decision in this library, and it was made to optimise a national distribution rather than to be defensible one student at a time. The lesson is that consequence sets the standard: a system this important needed each grade to be individually justifiable, its impact across groups measured and published before results day, and its workings genuinely open to outside scrutiny. The U-turn was the proof. A set of grades that cannot be defended one by one is not a fairer standard, it is a spreadsheet, and the moment each student asked why, the model had no answer.
The reference implementation of ConstraintGate, MetricRecord, and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Awarding GCSE, AS, A level, advanced extension awards and extended project qualifications in summer 2020: interim report (Ofqual, GOV.UK)
- Grading gaps in summer 2020: who was affected by differences between centre assessment grades and calculated grades (Ofqual evaluation, GOV.UK)
- Review of the approach to developing statistical models to award 2020 exam results (Office for Statistics Regulation)