Our full consultation and student survey covering proposals to change the review and appeals system for GCSEs, AS and A levels closed on Friday 11 March. Ofqual's Associate Director of Communications, Chris Shadforth, takes a closer look at some of the numbers.
We were delighted with the number of responses we received and we are now analysing them. Following some headline-grabbing comments in the papers, it was pleasing to hear the Chair of the Headmasters' and Headmistresses' Conference – speaking on LBC’s breakfast news* – agree with our Interim Chief Regulator, Amanda Spielman, that there is often no single ‘right’ mark in assessment:
Chris King (HMC): …“[Amanda Spielman] is of course right, if you can have a variation on an English paper from 17 to 19 then that's a band of subjectivity that could be easily understood.”
However, Mr King added:
“But with remarks we're not talking about a difference between 17 and 19, we're talking a difference between 7 and 19 in many cases.”
But is Mr King right? These charts show the frequency of mark changes following all marking reviews in GCSE English and GCSE maths in 2015. They show that in many cases no change in mark occurred at all, and where a change did occur it was typically 3 marks or less – broadly consistent with his own ‘band of subjectivity’.
But what about a 12 mark change, similar to the one Mr King suggests exists in ‘many cases’? Well, the data show that 137 GCSE English marks were changed by 12 marks or more in 2015. And 4 (four) GCSE maths marks were changed by that amount. That’s out of a combined total of nearly 100,000 reviews in these two subjects – or 0.14%. The bottom line is that the vast majority of marks that are reviewed are either left unchanged, or change by only a small amount – and it is misleading to suggest otherwise.
...
* LBC Morning News transcript
06:55, Friday 18 March 2016
Lisa Aziz (Host, LA): Amanda Spielman (AS), what is so wrong with the current appeals system that makes you want to change it?
AS: The problem with the current system is that it is not entirely fair because it gives an advantage to people who appeal over people who don't appeal. So, the problem at the root of this is that education measurement isn't quite as precise as people would like it to be. It's a bit like medical testing, two doctors looking at the same test results can come to different views and in the same way two good markers – who'll be teachers – can take different views about what is the right mark for a piece of work. One marker might give an English essay 17 marks and another 19 marks and neither of them is wrong. So we have to build a system around recognising that fairness does not mean absolute precision.
LA: So how might it change in practice then? What hurdles will students face in future if they want to challenge their grades?
AS: They'll actually face fewer hurdles because our proposal is that schools should be able to see the marked script – they usually get scanned in – so that the schools will be able to see if a mistake has been made, challenge it immediately and get it corrected. There's no obstacle. But the difference will be, if the first mark is reasonable then that mark will stand even if another marker might have given it a slightly higher mark.
LA: Chris King, let me give you my own example. My own son sent back about 5 AS module papers for remarking last year…and most did come back with different marks. Now we were really grateful for that appeals process because it did absolutely directly affect his UCAS entry. Do you have fears over examples like that changing?
Chris King (HMC Chairman, CK): Oh absolutely. I have very great fears of it changing. Amanda Spielman has referred to the concept of reasonableness. Reasonableness is dumbing things down to the lowest possible common denominator for an appeals process. What we are appealing for is an appeals process that is transparent and fair for all candidates. So that's why we have been working with the NAHT – 30,000 schools leaders are joined together on this one – absolutely determined that there should be a system that is fair for every single candidate. So they ideally get the mark that is correct the first time round, and if not there is an absolutely transparent appeals process.
LA: Amanda Spielman does it perhaps also come down to – examiners are facing too many calls for these remarks, so the attitude might be, 'do you know what, let's just make the whole process harder to get away from that’?
AS: No that is completely wrong, and what Chris said gets to the essence of it. He's talking about it as if there is one correct mark. It is a really difficult concept, but it is so important to understand the difference between marking mistakes and the variability that we can never eliminate. The concept of reasonable – the point is to pin it down in the context of a given subject to a given type of assessment. The margin of reasonableness is not the same in every case. We are absolutely not pushing for a low standard. We are pushing for the highest standards of marking that can be put in for any given type of assessment. And that's the point that I don't think has been properly understood.
LA: Chris why don't examiners deserve more of our trust?
CK: Well I think we have got to be fair to the examiners in total. There are very many good examiners, it's just that too many children end up with the wrong mark. 90000 papers regraded. That's 90000 people who are going to struggle in many instances to fulfil their dreams and we have got to have a recalibration of things. What we've got is Amanda appealing for a process driven approach to appeals so that as long as papers have been marked and we've followed through correctly at different stages that would be absolutely fine. She's of course right if you can have a variation on an English paper from 17 to 19 then that's a band of subjectivity that could be easily understood. But with remarks we're not talking about a difference between 17 and 19, were talking a difference between 7 and 19 in many cases. And to remove things like the code of practice and to say that reasonableness over fairness is a better approach is just simply wrong.
LA: Goodness this is a difficult one isn't it. We appreciate your time this morning. Thank you.
...
Our full response to statement issued by HMC/NAHT following consultation about marking reviews, appeals, grade boundaries and Code of Practice