Why did Johnny and Jenny's exam grades yo-yo over the summer? Here's some of the code behind UK results chaos

UK exams body Ofqual has published the code behind the summer's results fiasco. Sort of.

Like a 1980s hobbyist keenly keying in source from Home Computing Weekly and hunting for that sneaky bug or typo, you too can now peer at reams of code and wonder just how it all went so wrong. Assuming the statistical language R is your thing.

Presented in GitHub, the aim is to promote transparency. A laudable goal, if it weren't for the fact that, as the repo makes clear, "the code developed and presented here was not the final code used." It was up to exam boards to determine that. All Ofqual did was flash the source to demonstrate "how Ofqual's relevant regulatory requirements could be implemented."

Another fly in the ointment is a lack of data to run through the algorithms in order to enjoy a repeat of the summer's fireworks from the comfort of one's own computer. Ofqual is working with Ofsted, the Department for Education, and UCAS to make available the data used in 2020, but for now one will have to build one's own .csv files.

Even so, the R code is relatively easy to follow, with the key steps that built the historical performance and generated the predictions clearly visible in the source.

As a reminder, summer's fiasco was a result of the UK government opting to cancel the summer 2020 GCSE, AS, and A-level exam series in light of the COVID-19 pandemic. However, students still needed grades to progress to higher education, work, and so forth. Teachers were instead asked to submit the grades they reckoned their students would have achieved in the form of a Centre Assessment Grade (CAG).

Because teachers might be prone to optimism, those CAGs were standardised using Ofqual's preferred model, the Direct Centre Performance model (DCP), which would adjust things based on the historical performance of the college.

Those initialisms are important, and crop up in the code quite a bit.

Infamously, those calculated grades (which sparked protests after a good portion of students were downgraded) were abandoned and the teacher-predicted results used instead.

Guy Nason, chair of statistics at Imperial College London,welcomed the publishing of the source, declaring: "It's really good they've done this; we're on the way to transparency," although he noted that it would have been interesting to see what was in those "final codes". He also pointed out that communication in and out of the code "appears to be via spreadsheets" with a suitably scowly emoji.

As well as seeking some realistic sample data, Nason made a few of what he called "nerd points", observing it was "weird that they've essentially redefined the round function from R. Why? At first sight, seems to be written correctly, but why do it?" He added: "There's an interesting line in the DCP modelling code, where marks are removed from the cut point calculation where candidates had more than one mark from a previous year."

"It makes me wonder what else is in the code that isn't described in the written spec?"

While Ofqual is keen that this should be taken as a learning experience, the COVID-19 pandemic is continuing to disrupt education in the UK (and around the globe). As such, a look at what informed the thinking of 2020 would be useful ahead of whatever happens in 2021. ®