When asked to think of examiners marking scripts, many people still tend to think of them sitting at a desk with a pile of envelopes from schools and colleges, gradually working through each one in turn. In some subjects, that is still what happens, but it’s no longer the norm.
These days, the 63,000-strong cadre of examiners – most of whom are experienced teachers – do most of their marking on-screen. This use of technology has many advantages, which I set out below. But what does on-screen marking mean for schools, colleges and students?
Principally it means that there is no single examiner marking a school’s scripts, as with paper-based marking. Instead, the scripts are sent electronically to many different examiners. And where the marking takes place at item (question) level, one school’s scripts (and each student’s script) could be marked by many different examiners. Such a diffuse system might be difficult to picture, but it does have advantages.
How does on-screen marking improve the quality of marking?
On-screen marking has several advantages over paper-based marking, including:
- reducing the risk of clerical errors, because the marking system adds up the item marks
- reducing the risk of scripts being lost in the post, as they only make one journey from the school or college to the scanning centre
- giving the exam boards better information on marking progress, because they can see how many scripts are being marked on a daily basis
Crucially, it also allows better quality assurance of marking, while markers are actually marking.
Paper-based marking processes
With paper-based marking, examiners send samples of their scripts to their team leader or principal examiner. They send one sample very early on, and the team leader checks this before the examiner is cleared to continue marking. And they send another sample about halfway through the marking. For both samples, the team leader reviews the marking to make sure it’s in line with the agreed standard. Other checks may also be employed by exam boards, and in all cases it is for them to determine appropriate action, including stopping them from marking and passing the work to other examiners.
On-screen marking processes
With on-screen marking, exam boards use several different approaches. Where scripts are split up into separate items (questions) and marked, most use ‘seeding’, although some use double marking (more on this below). After the exam has taken place and before the marking starts, senior examiners review student responses and select a number of ‘seed’ items. They then agree a definitive mark – the mark that the response will get and that they all agree is the most appropriate - for each of these seed items, and agree a marking tolerance. A 1-mark item would likely have no tolerance – if the response is worthy of the mark, examiners would be either right or wrong in their marking. In contrast, an 8-mark item might have a tolerance of plus or minus 1 mark – if the definitive mark is 6, examiners awarding 5, 6 or 7 marks would be judged to be marking within tolerance. This is not just for administrative convenience, but reflects that there may be legitimate differences of opinion between markers, and is also the case with paper-based marking.
In some systems, every time an examiner logs on to mark, they will have to mark some of these seed items and their marking must be within tolerance before the system will allow them to start marking formally. Examiners also mark these seed items (without knowing that they are seeds) as they do their marking, and if their mark is out of tolerance when compared with the definitive mark for that seed item, they will be temporarily stopped from marking that particular item (although they can continue marking responses to other questions). Where that happens, the team leader will review the examiner’s marking and provide feedback on his or her marking. If an examiner is repeatedly stopped on a particular item, they may be permanently stopped from marking that item and the work passed to someone else.
Not all on-screen marking is done by item. In some subjects, examiners mark a whole script on-screen. This tends to be in subjects where students are writing longer, essay-style responses. In such cases, exam boards often use seed scripts but they sometimes also use double marking to check the marking quality. Examiners mark as normal but some of their marking is automatically sent to a team leader or another examiner for marking (without them knowing the mark the first examiner awarded). If the two marks differ by more than an agreed tolerance, a third marker ‘adjudicates’ the mark. Examiners receive feedback on their marking, and those who are repeatedly flagged as out of tolerance may be stopped from marking.
Increased objectivity
One of the benefits of the on-screen marking process is that the quality checks are less subject to bias. Where team leaders are reviewing the marks that examiners have given, they are more likely to agree with the marks awarded. Where examiners’ marks are compared to the definitive mark for seed items, or where two examiners separately mark an item and their marks are compared, as in on-screen marking, the checking is more objective, and more likely to result in more consistent marking overall.
With all that I have said, you may question why paper-based marking is still used. In some subjects, on-screen marking is not feasible because of the nature of the work produced, or the relatively small number of entries means it is not worthwhile producing seed items, for example.
So, given on-screen marking is now the norm, it’s worth remembering what this means come August; each student may get a single final mark per qualification, but for most subjects it will have come about through the combined effort, experience and dedication of many different examiners.
3 comments
Comment by Francis bramhill posted on
No references made to the health and safety aspects of continued online marking that examiners are exposed to after marking in this manner.
Comment by Patrick Radcliffe posted on
Being a retired teacher who has marked various English series for AQA for 21 years, mostly twice a year, I found this blog interesting and in a sense comforting.
It shows a system which gives fairness to each and every candidate. That, for me, is what any examination should provide.
Comment by Charlie posted on
plz make the grade boudries low x