Casebook

England's 2020 exam grades

The one case in the casebook where correction was fast, which is what makes it useful. The model's accuracy for any one student was never shown. No appeal let a student argue it was wrong about them. The halt worked only because it came before anyone had come to depend on the grades.

System
A standardization model that replaced canceled A-level and GCSE exams by fitting each school's historical grade distribution to its current cohort, overriding teachers' assessed grades for all but the smallest classes.
Where
England · Ofqual, Department for Education
When
Aug 13 to 17, 2020
Scale
About 39% of A-level grades issued below the teacher-assessed grade; the effect fell hardest on large cohorts in state schools and lightest on small classes, which were exempt.
Time to halt
Four days
Errors carried by
Students in large cohorts at state schools, where the model overrode teachers' grades most. They could appeal only through their school, and not on the ground that the model had ranked them wrongly. Small classes were exempt.
Halted by
The Secretary of State, after Scotland had already reversed its equivalent and universities had begun allocating places on the model's grades.
After the failure
Process unchanged

The record

What happened

From the primary sources listed at the end. Where a finding is quoted, it is the inquiry's, not ours.

Timeline
  1. 2020 Exams are canceled. The Secretary of State directs Ofqual to keep grades broadly similar to previous years.
  2. Aug 11, 2020 Scotland withdraws its equivalent model.
  3. Aug 13, 2020 Results are issued. Ofqual's own report says the model cannot be validated for individual students. No appeal lets a student argue it ranked them wrongly.
  4. Aug 17, 2020 Four days after results, Ofqual and the Department for Education announce that teachers' grades will stand.
  5. Mar 2021 The statistics regulator finds the failure was in how the choices were made, explained, and opened to challenge.

When the 2020 exams were canceled, the Secretary of State directed Ofqual to award grades and to ensure the distribution was broadly similar to previous years. Schools submitted a centre-assessed grade and a rank order for each student. Ofqual's model then fitted each school's grade distribution from the previous three years, adjusted for the cohort's prior attainment, to the rank order. Where a subject cohort at a school was five or fewer, the teacher's grade was used as submitted; between five and fifteen, a blend.

Results were issued on August 13. The regulator's interim report published the same day acknowledged that the model could not be validated against outcomes for individual students, since none existed, and that its aggregate agreement with past distributions was the design target rather than a measurement of accuracy. Students could appeal only through their school, on grounds of administrative error or, under a policy announced two days before results and withdrawn two days after, a higher mock grade. There was no ground of appeal that the model had ranked the student wrongly.

Scotland's regulator had already withdrawn its equivalent model on August 11. On August 17 Ofqual and the Department for Education announced that centre-assessed grades would stand. The Office for Statistics Regulation's review, published in March 2021, concluded that the model's limitations were understood by those building it and that the failure was in how the choices were made, explained, and opened to challenge.

Scoring

The six safeguards

One finding per safeguard, with the clauses it falls under and the law it applies.

  1. held

    Capability

    What could the assembled system do?

    The model did what was asked of it: it reproduced the prior distribution. Its capability was declared, its exclusions were declared, and its behavior on small cohorts was published in advance. Capability is not where this case failed.

    STD-06 §1.2 Law I

  2. drifted

    Authority

    Which actions was it permitted to take, for whom, until when?

    The authority came from a ministerial direction to hold the distribution steady. That direction was itself the policy the model implemented, and it was never re-examined once the model's effects on individual students were visible. A direction that sets a limit for the whole cohort says nothing about what may be done to any one student.

    STD-07 §2.2 STD-08 §2.1 Law II

  3. failed

    Evidence

    What evidence justified that authority?

    The model was checked against whole-cohort results and then used to grade individual students. Ofqual's own report said it could not be checked against any one student's result, since none existed. No one checked it where the harm happened: one student's grade.

    STD-06 §2.2 STD-06 §2.4 Law III Law X

  4. held

    Dependency

    How hard had withdrawal or substitution become?

    The halt was thrown four days in, before universities had finalized admissions and before the grades had been used for anything irreversible. Dependence had not yet set. Three weeks later, with places confirmed and courses full, the same reversal would have been impossible in practice. The clock, not the decision, is what made this case recoverable.

    STD-06 §5.5 Law V Law XI

  5. failed

    Standing

    Who could challenge a decision, and did the challenge have to be answered?

    Students could not appeal on the ground that the model had got them wrong. Appeals ran through the school, on grounds that excluded the decision itself, and the one ground that might have helped was announced and withdrawn within four days. The students who bore the errors had no way to question the model itself.

    STD-02 §8.1 STD-02 §8.2 STD-02 §8.4 Law VII

  6. drifted

    Correction

    Which interventions were still possible, and how fast?

    The halt existed and was thrown. It was not the operator's; it was ministerial, taken under public pressure, and after a neighboring jurisdiction had gone first. A correction that works when a minister chooses to use it is real, but it is not a halt the operator had tested in advance, with a named owner and a threshold set beforehand.

    STD-06 §1.3 STD-06 §2.2 Law IV Law IX

Remediation

Avoid this failure in your system

The diagnostic that tests for this defect, the standard that would have required a fix, and the essay that explains why it recurs.

1. Run the diagnostic

Delegation audit

Asks, for one workflow, who authorized each action, on what evidence, for whom, and until when, and whether anyone can halt it.

2. Apply the standard

STD-06: Human impact safety case

Requires the operator to set each test's threshold before the test runs, and to say what happens when it is crossed. An error rate clearly worse for any one group than for others counts as crossing it.

3. Read why it recurs

The model of a person is not the person

Why does a model that is right about most students not get to be final about any one of them?

After the failure

Did the institution change?

Whether the institution changed the rule, category, or workflow that produced the errors, or corrected the errors and left that process as it was.

Process unchanged

The decision was reversed — every model-based grade was replaced within four days — but no institution's machinery was revised by the failure. The reversal changed the operative decision rule for the affected cohort: model-based grades were replaced, so the halt was more than a reversal of individual decisions. What did not change was the institution's machinery. Ofqual and the Department for Education withdrew the model rather than correcting the rule that generated the harm, the practice of checking a model against whole-cohort results and then applying it to individual students was never revisited inside the institution, and the subsequent approach moved to teacher judgment. A minister ordered the halt under public pressure after a neighboring jurisdiction had gone first — an emergency reversal, not a halt the operator had set in advance. The office that reviewed the episode was the statistics regulator, external to the operator.

Read the theory: exception learning · Corrective learning eval suite

The record that was not there

Every case reduces to a record the standards require and the operator did not hold. This is that record.

A threshold set in advance, with the action that follows if it is crossed. Under STD-06 §2.2, that means saying before release what share of downgrades, or what gap between school types, would stop it. Had that number existed, the halt on August 17 would have been a planned halt, not a reversal under pressure.

The primary record

Courts, statutory inquiries, regulators, and parliaments, in that order of weight. Operator and NGO documents appear only where they record what the primary record established. No press coverage.

  1. Regulator Ofqual, Awarding GCSE, AS, A level, advanced extension awards and extended project qualifications in summer 2020: interim report · Aug 13, 2020
  2. Regulator Office for Statistics Regulation, Ensuring statistical models command public confidence: lessons learned from the 2020 A level and GCSE grades · Mar 2, 2021
  3. Parliament House of Commons Education Committee, Getting the grades they've earned: Covid-19: the cancellation of exams and 'calculated' grades · Jul 11, 2020