Casebook

England's 2020 exam grades

The one case in the casebook where correction was fast, which is what makes it useful: a model with no evidenced accuracy at the level of the individual it was applied to, no appeal on the ground that it was wrong about that individual, and a halt that worked only because it was thrown before dependence had set.

System
A standardisation model that replaced cancelled A-level and GCSE exams by fitting each school's historical grade distribution to its current cohort, overriding teachers' assessed grades for all but the smallest classes.
Where
England · Ofqual, Department for Education
When
13 to 17 August 2020
Scale
About 39% of A-level grades issued below the teacher-assessed grade; the effect fell hardest on large cohorts in state schools and lightest on small classes, which were exempt.
Time to halt
Four days
Halted by
The Secretary of State, after Scotland had already reversed its equivalent and universities had begun allocating places on the model's grades.

The record

What happened

From the primary sources listed at the end. Where a finding is quoted, it is the inquiry's, not ours.

When the 2020 exams were cancelled, the Secretary of State directed Ofqual to award grades and to ensure the distribution was broadly similar to previous years. Schools submitted a centre-assessed grade and a rank order for each student. Ofqual's model then fitted each school's grade distribution from the previous three years, adjusted for the cohort's prior attainment, to the rank order. Where a subject cohort at a school was five or fewer, the teacher's grade was used as submitted; between five and fifteen, a blend.

Results were issued on 13 August. The regulator's interim report published the same day acknowledged that the model could not be validated against outcomes for individual students, since none existed, and that its aggregate agreement with past distributions was the design target rather than a measurement of accuracy. Students could appeal only through their school, on grounds of administrative error or, under a policy announced two days before results and withdrawn two days after, a higher mock grade. There was no ground of appeal that the model had ranked the student wrongly.

Scotland's regulator had already withdrawn its equivalent model on 11 August. On 17 August Ofqual and the Department for Education announced that centre-assessed grades would stand. The Office for Statistics Regulation's review, published in March 2021, concluded that the model's limitations were understood by those building it and that the failure was in how the choices were made, explained, and opened to challenge.

Scoring

The six state variables

One finding per variable, with the clauses whose obligation it falls under and the law it instantiates.

  1. held

    Capability

    What could the assembled system actually do?

    The model did what was asked of it: it reproduced the prior distribution. Its capability was declared, its exclusions were declared, and its behaviour on small cohorts was published in advance. The capability variable is not where this case failed.

    STD-06 §1.2 Law I

  2. drifted

    Authority

    Which actions was it permitted to take, for whom, until when?

    The authority came from a ministerial direction to hold the distribution steady. That direction was itself the policy the model implemented, and it was never re-examined once the model's individual-level effects were visible. Authority that names its ceiling as an aggregate says nothing about what it may do to any one person.

    STD-07 §2.2 STD-08 §2.1 Law II

  3. failed

    Evidence

    What propositions justified that authority?

    The model's accuracy was evidenced at the level of the distribution and applied at the level of the student. The regulator said as much in its own report. A proposition about a cohort was used to justify an action against an individual, which is the gap Law X names: the eval sat one layer below the harm.

    STD-06 §2.2 STD-06 §2.4 Law III Law X

  4. held

    Dependency

    How hard had withdrawal or substitution become?

    The halt was thrown four days in, before universities had finalised admissions and before the grades had been used for anything irreversible. Dependence had not yet set. Three weeks later, with places confirmed and courses full, the same reversal would have been operationally impossible. The clock, not the decision, is what made this case recoverable.

    STD-06 §5.5 Law V Law XI

  5. failed

    Standing

    Who could challenge a decision, with what procedural force?

    Students could not appeal on the ground that the model had got them wrong. Appeals ran through the school, on grounds that excluded the decision itself, and the one ground that might have helped was announced and withdrawn within four days. The party bearing the error had no route to put the delegation in question.

    STD-02 §1.3 STD-02 §3.1 STD-02 §8.1 Law VII

  6. drifted

    Correction

    Which interventions stayed feasible, and on what clock?

    The halt existed and was thrown. It was not the operator's; it was ministerial, taken under public pressure, and after a neighbouring jurisdiction had gone first. A correction that works when a minister chooses to exercise it is real, but it is not a rehearsed control with a named owner and a declared threshold.

    STD-06 §1.3 STD-06 §2.2 Law IV Law IX

Learning outcome

Where the failure trajectory ended

Handled or learned: did the sequence from failure to challenge end in case resolution, or did the exception change the machinery that produced it?

absorbed — exception absorption versus exception learning

The decision was corrected — every grade was reversed within four days — but no institution's machinery was revised by the failure. Ofqual and the Department for Education withdrew the model rather than repairing the rule that generated the harm; the evidentiary practice of validating an aggregate and applying it to individuals was never revisited inside the institution. The halt was thrown by a minister under public pressure after a neighbouring jurisdiction had gone first, which is a correction of the case, not a change to the rule. The office that reviewed the episode was the statistics regulator, external to the operator.

Read the theory: exception learning · Corrective Learning eval suite

The missing record

The one object that was not there

Every case reduces to a record the standards require and the operator did not hold. This is that record.

A declared threshold with the action that follows a breach. STD-06 §2.2 asks the operator to say, before running the model, what share of downgrades or what disparity between school types would stop the release. Had that number existed, the halt on 17 August would have been a control firing rather than a reversal under pressure.

Sources

The primary record

Courts, statutory inquiries, regulators, and parliaments, in that order of weight. No press coverage.

  1. Regulator Ofqual, Awarding GCSE, AS, A level, advanced extension awards and extended project qualifications in summer 2020: interim report · 2020-08-13
  2. Regulator Office for Statistics Regulation, Ensuring statistical models command public confidence: lessons learned from the 2020 A level and GCSE grades · 2021-03-02
  3. Parliament House of Commons Education Committee, Getting the grades they've earned: Covid-19: the cancellation of exams and 'calculated' grades · 2020-07-11