https://ofqual.blog.gov.uk/2017/02/03/grade-boundaries-the-problems-with-predictions/

Grade boundaries: the problems with predictions

Would you believe someone who told you now that they could predict, to 1 degree Celsius, the temperature on the day your students get their GCSE results this summer? Perhaps not. You might expect they could get reasonably close, based on previous average temperatures, but the British summer is infamously unpredictable.

We know that the big question for schools this year is where the grade boundaries – the minimum mark required for a grade - will be set in the new GCSEs in English and maths. It’s clear why this information would be helpful: to predict likely achievement, to motivate students, to report to parents and others.

Exam boards are not predicting the boundary marks, and are rightly urging caution. Other organisations, responding to teacher requests, are far less cautious. Some organisations have had their member schools sitting their own mock exams and have provided ‘results’ and ‘grade boundaries’ on the basis of that exercise. That’s really helpful, yes?

Actually, no.

There are many good reasons to be cautious ahead of 2017. Here are our top three.

  1. Even in well-established qualifications, grade boundaries are never set in advance. And for good reason. It’s almost impossible to predict precisely how much easier or more difficult students will find a paper compared to previous years. Even the examiners who write the papers find it challenging. So exam boards wait until the students have taken the exam, compare their performance to that of previous cohorts, and then set the grade boundaries. 2017 is no different.
  2. 2017 sees the first live exams of new GCSEs in English language, English literature and maths. Exam boards have issued specimen papers and some sample answers, but this will be the first time these new style papers are taken by students and so we should be cautious in speculating about where the grade boundaries might be set.
  3. Statistics will play a key role in making sure this year’s students are not disadvantaged by being the first to sit these new GCSEs. Exam boards will use prior attainment at Key Stage 2 for the 16-year-old cohort to predict likely achievement at the key grades – 1, 4 and 7. The bottom of these grades will be aligned with the bottom of grades G, C and A respectively so the proportions of students achieving these grades or higher will be broadly similar to the previous year. We, and the exam boards, will have the full national picture; other organisations will only have a sub-set of the cohort, which may not be representative of the national situation.

Our advice to schools would be this: do not rely on any predictions of grade boundary marks for new GCSEs next summer. They are only a best guess, regardless of any modelling that might have been done. If the boundaries in the summer turn out to be different, which is quite likely for all the reasons set out above, you and you students might be disappointed.

So what can schools rely on? Well, we’ve already said that exam boards will set standards so that broadly the same proportion of students will achieve a grade 4 and above as previously achieved a C and above. Similarly, the proportions achieving grade 7 and above will be broadly the same as those achieving the old grade A and above. And we’ll be basing this on 16-year-old students.

What does that mean in terms of numbers? In 2016, in English and in maths, about 70% of 16-year-old students achieved a grade C or above. So we’d expect a similar percentage to achieve a 4 and above in 2017. At A, the figures are different in English and maths: in 2016 16% of 16-year-olds achieved an A or above in English and 20% achieved an A or above in maths. Again, we expect these figures to be broadly the same in 2017.

Our priorities during any series are that exams are delivered to plan, results are issued on time and are accurate, and that standards are maintained. We can’t tell you what the temperature will be on 24 August this year, but we can tell you that about 70% of 16-year-old students in England will achieve a grade 4 or above. I hope this gives you reassurance as you and your students continue your preparations for this summer.

55 comments

  1. D Morgan

    Schools are being judged on 5 and above, not 4. I think the expected % of 5 and above would be far more important and is what schools are worrying about!

    Link to this comment Reply
    • Cath Jadhav

      Thanks for your comment.

      We know that schools are concerned about grade 5 but we think there's good reason to be cautious in trying to predict the percentage of students that will achieve grade 5 or above. Grades 7, 4 and 1 will be set using predictions (and grade 9 will be set using the agreed formula). The other grades will be set arithmetically - in the case of 5, exam boards will divide the number of marks between 4 and 7 by three (adding any remainders to the lower grade width). Predicting the percentage that might achieve a 5 or above is more difficult because we don't yet know whether the distribution of marks in the new GCSEs will be similar to the previous GCSEs.

      Link to this comment Reply
      • r holmes

        yet again sight of what matters has been lost. Students want to know what they need to do to get to the point where they can be assured of progressing to their chosen pathways. yet if only 70% are to be awarded the grade 4, 30% will not even if every child in the country achieved above the 2016 attainment expectations for all ubjects.
        Glass Ceilings and moveable success criteria. Give the educators what they need, grade boundaries . allow any one to ACHIEVE!

        Link to this comment Reply
      • Jacqui

        How will this work in the Foundation Maths?

        Link to this comment Reply
    • Richard Warren

      In Eng & Maths, for this year only, schools will be judged on 4 and above. Judgements on 5+ only apply from 2018 onwards.

      Link to this comment Reply
      • acreen

        Richard Warren, that's not correct. Your confusion is over funding rules for post-16. For 2 years students with a 4 do not need to resit, but those with a 3 or below do in order for the institution to receive funding. From 2019 this changes to a 5. But for school accountability measures, 5 will be a good pass from 2017, never a 4.

        Link to this comment Reply
      • Peter Mattock

        Richard - the 5+ measure for schools is immediate. The 4 for the next two years is to do with post 16 funding and has nothing to do with school accountability.

        Link to this comment Reply
    • Louisa Wilson

      How will the statistics work for those of us still doing the old GCSE? It will also be an unusual cohort. I'm in FE with a mixed bag of 16-19 year olds, adults and international students. It's hard to imagine how the like for like comparison will work. Has there been anything written about this?

      Link to this comment Reply
  2. Paul Hammond

    Cath - the PIXL sample size is over 100,000 so that is statistically valid. The stakes are so high with these new exams that schools want to get it right - for themselves as well as for the students. I can't see what is wrong with applying your own methodology to the outcomes and giving schools some indicative scores with which to inform their professional judgements - particularly regarding tiering decisions over Maths.

    Link to this comment Reply
    • Peter Mattock

      Paul - unfortunately the PiXL papers are not at the standard of live exams and do not meet the criteria that are set down to the boards for paper creation. So although the pupils might be representative, the papers are not and therefore neither are the boundaries.

      Link to this comment Reply
      • Paul Hammond

        They don't claim to be, but in the absence of any other resource, they are close enough to be of use to schools. If they were not,you wouldn't have 100,000 students sitting them.

        Link to this comment Reply
  3. Peter Mattock

    Unfortunately, we have to have something. Colleges require predictions for basing their offers, and schools require predictions and targets. In the absence of any further information practitioners do what they can; even if we know it is only best guess.

    Link to this comment Reply
  4. Fran

    Do you imagine that parents and employers understand this? There are school leadership teams who haven't got their heads around it. When will the public education campaign start?

    Link to this comment Reply
  5. John Lloyd

    Cath,

    Thanks for your post. I think that most teachers understand the inherent dangers in predicting grade boundaries however under pressure from SLT, parents and, let's not forget, students, most parties want to know what grade they are working at, especially after trial exams.

    However, putting that to one side, and speaking as a maths teacher, what would be useful is some guidance of what you expect the broad proportions to be of students entered for each tier. I understand from what I've seen elsewhere that you believe there is a low risk of large proportions of students being entered for the Higher tier: what does your research indicate the proportions of Higher/Foundation entries might be please?

    This information would be extremely useful to teachers who are finding the tiering decision for students at the moment very difficult.

    Thanks,
    John

    Link to this comment Reply
  6. Stephen

    The problem is not the grades. The problem is deciding on higher or foundation paper for each individual student. What mark do they need to be getting on the higher paper to be sure of a 4 or better? We knew that reasonably well. Now we don't know. Your article works fine from a national position. But for individual schools, making decisions on individual students, we need clearer advice. Otherwise students will be entered for the wrong tier of paper and not get the results they deserve.

    Link to this comment Reply
    • Ash

      An Edexcel representative told me that by April of their GCSE year, if a student cannot gain 20 out of 80 marks on a higher paper, they need to be entered for Foundation…

      Link to this comment Reply
      • Stephen Fuller

        I think 20 will easily be a 4. Top 50% of paper is aimed at 789 grades. So 41/80 should be a 7. 31 might be a 6, 21 a 5?

        Link to this comment Reply
  7. Emma

    Why, if a 4 is considered a C are schools being judged on 5? This is the most confusing element of all of this?

    Link to this comment Reply
    • Peter Mattock

      It is where the govt suggests pupils need to be to compare well with other pupils internationally, and so they are interested in % of pupils meeting this standard.

      Link to this comment Reply
      • David Coughlan

        My problem with this is that the Government, through Ofqual and the exam boards, will set the percentage that achieve a 5 or above. So how can schools really change that at the cohort level?

        Link to this comment Reply
  8. Mel Muldowney

    Yes, we are worried about 5's - wouldn't you be if that was what you got measured on?

    However of bigger concern right now is the damage being done to teacher stress/morale because of the tiering decisions that are being made at the minute. Having a third consultation about grading including the ability to move the grade 3 on higher tier (outcome still not published BTW) at such a late point in the day hasn't helped either.

    On a more positive note ... thanks for the blog post.

    Link to this comment Reply
  9. Debra Kidd

    The difference between the weather and the exam system is that you don't know in advance what the weather will be like - it hasn't been written, set, checked, approved and applied in advance. Whereas an exam, in theory, has an established, written set of criteria which, in theory should give a reliable set of expectations. The only reason you can't (or won't) allow for predictions is because you adjust the grade boundaries to ensure that, whatever their achievements - whatever they've done in terms of meeting the criteria in the specification, you'll adjust the final pass rates to ensure that the same number roughly pass at the same rate every year. This means that no year group can be better than another, no matter how hard they work, how much more experienced their teachers become. This system works against every principle of growth mindsets. It means that schools measured in terms of progress are stuffed - they can't possibly get better without someone else getting worse. It's a rigged system, designed to protect the appearance of consistency over effort.

    Link to this comment Reply
    • Gordon Spitz

      This is an important argument - if pupils met criteria such as 'excellently detailed and focused ideas' for example, they could still end up with a level 6 or 7 and that makes no sense. I have seen this happen already with controlled assessments and coursework at the bottom end, someone who is more than competent ends up with a 4 they can do nothing with!

      Link to this comment Reply
  10. Brian Lightman

    So holding schools to account for five and above is perhaps something the govt needs to be cautious about! Is that your advice to them?

    Link to this comment Reply
  11. What are we doing to our students?

    What assurances can that yourselves can give us that a student who achieves a grade 5 at foundation has shown they are better than a student who avhieves a grade 4 at higher?

    Link to this comment Reply
  12. Paul Hammond

    Schools are keen to know where they and individual students are against national benchmarks. The stakes are made very high for both parties. The sample size for PIXL is over 100,000 and through analysis of Ks2 scores, matched the profile of the national cohort.

    Link to this comment Reply
    • Liam Wilkinson

      One issue I would imagine with the PIXL sample is that it does not include a great many of the high attaining schools in the sample which would skew it.

      Link to this comment Reply
      • Paul Hammond

        Liam - that is not the case. PIXL asked schools for the ks2 scores of their entered students and the distribution matched the national cohort - for all abilities. It's also an inaccurate assumption that PIXL is only used by schools in the lower half of the performance range.

        Link to this comment Reply
        • Stephen

          Assuming that everyone has KS2 scores. Which many private school students don't. If Pixl grade boundaries are right I'll be in clover, but they won't be. The exam also was not as hard as other sample papers.

          Link to this comment Reply
  13. Granny B

    That still doesn't help then? As the first comment says, it's the 5 we are all interested in. It's unfair to say we will be judged on 5 then only reference the 4 in explanations.

    Link to this comment Reply
  14. Winston Ellis

    Hello Cath. First of all thank you for you further clarification on this, it has gone some way to reassuring me that my approach so far is been correct. Though this may somewhat be controversial, I do agree with the changes that have been made to the Maths qualification as in the past I have been able to move pupils from U to grade E within a matter of months; which does not do the subject I love and I’m passionate much justice. I do however have issues with: the way it was introduced, the number of sample papers that were issued only to be told they were, not correct, too hard, too easy, easier/harder than another board etc. For a period of time it made us teachers feel that no real changes were going to be made and also discouraged us from making the necessary changes to our teaching to address the new curriculum.
    My only query really lies in the conundrum of correct tier of entry for the pupils who we are targeting the new grade 5. In the old Curriculum with the grade boundaries relatively stable over the last few years (with minor increases over time) we were fully aware of the correct tier of entry for each pupil. With the current status quo, without having additional guidance, many students will be entered for a tier that will not give them the best outcome. What guidance would you give?

    Link to this comment Reply
  15. Jon

    I cannot understand why the boards have been getting flak about not publishing indicative grade boundaries. They never have before and it is obvious that they can't. I guess it is because on previous occasions one could reasonably assume that those who would have got Cs will do again and now we have little idea what that means.
    The lack of many mock papers is the unforgivable thing.
    I don't agree with the idea that what the other organisations are doing lacks value though. It isn't about predicting grade boundaries but rather ranking your cohort against other comparable ones to see who is on course against a new syllabus and new assessment objectives.

    Link to this comment Reply
  16. Andrew Bruff

    Can you give advice on the percentage who will achieve a grade 9? It's been said before by OFQUAL that 20% of those achieving 7 or above will achieve a 9. If 17% are going to achieve a 7, that means 3.4% will achieve a 9. This is broadly similar to A* of old, but I was under the assumption that 9 is harder to achieve than A*?

    Link to this comment Reply
  17. Ash

    @Stephen

    Edexcel's advice is that a student should score at least 20 marks per Higher paper (so 60 in total) 'to be entered' for the Higher tier.

    Whether that's a slight indication towards what a grade 4 might approximately be (ie. ~60 marks or ~25% on the Higher paper) ... I'm not sure.

    Link to this comment Reply
  18. Winston Ellis

    Hello Cath. First of all thank you for you further clarification on this, it has gone some way to reassuring me that my approach so far is been correct. Though this may somewhat be controversial, I do agree with the changes that have been made to the Maths qualification as in the past I have been able to move pupils from U to grade E within a matter of months; which does not do the subject I love and I’m passionate much justice. I do however have issues with: the way it was introduced, the number of sample papers that were issued only to be told they were, not correct, too hard, too easy, easier/harder than another board etc. For a period of time it made us teachers feel that no real changes were going to be made and also discouraged us from making the necessary changes to our teaching to address the new curriculum.
    My only query really lies in the conundrum of correct tier of entry for the pupils who we are targeting the new grade 5. In the old Curriculum with the grade boundaries relatively stable over the last few years (with minor increases over time) we were fully aware of the correct tier of entry for each pupil. With the current status quo, without having additional guidance, many students will be entered for a tier that will not give them the best outcome. What guidance would you give?

    Link to this comment Reply
  19. Sally Scott

    Some of us have performance targets based on GCSE results. Knowing where students are is reassuring. Lick finger guessing is stressful enough that some of us want out of the profession. Grade 5 counts for school stats. Do not be fooled by other suggestions!

    Link to this comment Reply
  20. Tom Graham

    How is the country ever supposed to get better results if (using the above example) 30% ish of students will ALWAYS get a D/3 or lower?

    Link to this comment Reply
  21. Nicko

    Gove's legacy

    Link to this comment Reply
  22. Mel Muldowney

    Allowing that comment about schools being measured on a 5 is just irresponsible and WRONG!

    Link to this comment Reply
  23. Mrs LJ

    As a parent I understand what the government want for mathematics- however I am concerned for my son who unless he receives a level 5 cannot continue in the 6th form of his current school-- surely because of these changes to curriculum a level 4 should be acceptable until such time as the standard has been raised to 70% achieving a level 5 or above -a phased change in other words a period of adaptability for teachers for whom until now have taught maybe from a different perspective

    Link to this comment Reply
    • J

      No the new level won't increase, 30 percent will always fail then more when they increase to 5, even more will be failures! I assume we are trying to fill the lettuce fields post brexit otherwise the proles might revolt.

      Link to this comment Reply
  24. J

    Don't try to predict grades, but this is how you could predict grades?!?

    What worries me most about grade spread standardisation year on year?

    1) 30% always fail no matter the standard achieved

    2) that when some one @ the dfe realises 1) or agrees with me on 1) they will change the GCSEs back to the old system

    Link to this comment Reply
  25. Cath Jadhav

    Thanks to everyone who took the time to read this blog. The comments came in thick and fast over the weekend and they’ve focused on a number of key themes:

    1. Our focus on C/4 (and A/7 and 1/G) – when we made the decision to change the grade scale from letters to numbers, we wanted schools, colleges, parents, employers and everyone else who has an interest in these qualifications to have clear anchor points between the new and old grades. We set those anchors at 7/A, 4/C and 1/G. For example, if a college requires students to achieve at least a grade C to start an A level course, they would have a good idea of how that might translate into a number grade. Schools, colleges, employers, and others are, of course, free to set their own entry thresholds but we have been clear about the purpose of these anchor points.

    2. Tier entry in maths – there is overlap in maths at grade 4 and 5, and there is a ‘safety net’ grade 3 on the higher tier. This will be a narrower grade (in the sense of the number of marks) than other grades, because it’s intended for those students who just miss a grade 4.

    Some of you mentioned the more difficult decisions about tier entries this year, because of the changes. From your comments, it appears that there is plenty of advice out there, including from exam boards. Every student is different but it’s worth remembering the anchor point at 4/C and the overlap grades when making those decisions. We’ve interviewed a small number of schools on just this topic (and there’s a report on the way very soon) and they told us generally that they’re entering more of their students for foundation tier this year. The exam boards have mechanisms in place to align their standards across the tiers, and we’ll be monitoring that very closely in the summer.

    Some of you also referred to the recent consultation on the technical rules for setting the grade boundaries in the context of grade 3 on the higher tier in maths. I want to reassure you that what we’re putting in place is no different to the previous rules about setting the ‘safety net’ grade E on the higher tier.

    3. Reliability of mock exams – several comments queried our caution over the use of mock exam data. Yes, it is based on large numbers of students but the organisations involved do not know how representative it is of the national cohort, and that’s why we would urge caution in relying on grade boundaries derived from those mocks. And even mock papers set by the exam boards may be slightly easier/harder than the live exams in the summer, and students won’t necessarily have completed their courses when they took the mock papers: all reasons to treat any results/grade boundaries with caution.

    4. Grade 9 in the new GCSEs – there is more detail in our 9 to 1 newsletter on how grade 9 will be set in the first year of these new qualifications and I would encourage you to sign-up to this newsletter here, but there will be fewer grade 9s awarded than A*s in the previous qualifications.

    5. Schools/students not being able to improve – a few of you have commented that the ‘comparable outcomes’ approach means that students and schools can’t improve, but recent research from Cambridge Assessment shows that’s not the case. We’ve also introduced a national reference test to help us measure year-on-year progress; this year’s test will provide a baseline and in future years we’ll be able to compare performance with that baseline.

    6. ‘Old’ GCSEs – we haven’t forgotten that a large number of post-16 students this summer will be taking a final version of the old GCSEs in English and maths. In recent years the number of post-16 students re-sitting these GCSEs has increased (due to funding changes) and so exam boards have a good few years’ experience of setting standards for this cohort, including in the November series.

    Link to this comment Reply
  26. Mel Muldowney

    Regarding your comment about the consultation on grade 3 (1) when are we likely to see sight of the outcome? (2) what's to stop the powers that be using this ability to shift the grade 3 closer to a 4 ( rather than half the difference in marks between a 4 & a 5) so as to manipulate tiers of entry and make more people take the Foundation tier?

    Link to this comment Reply
  27. JH

    A great sadness here is that the system previous to GCSE qualifications, namely 'O' levels, used a norm referenced grading allocation process.

    This was discredited for exactly the concerns expressed in some response comments above ie. the pupil population, irrespective of any improvement in the standard of their knowledge and understanding, could not demonstrate that improvement, and neither could the schools those pupils were taught in. GCSE qualifications were introduced based upon a criterion referenced grading allocation through which it should have been possible to demonstrate year on year improvement, or decline.

    The government, and through its will the DfE, are about to foist a repetition of the follies of the past and yet appear blind to the fate they are assigning members of particular groups of pupils and the schools which, within their rolls, educate a preponderance of those pupils.

    Link to this comment Reply
  28. Mark Edgar

    In the current climate, where every grade in every subject for every student counts, it is VITAL to have grade boundaries. It is really unhelpful not to be able to tell students what grade they get in a mock examination. Edexcel collected data on their mock specimen papers, with a view to giving an idea of boundaries, then refused to do so - again most unhelpful. It is a mess with English and mathematics this year; next year with all subjects going over to the ridiculous 9-1 scale it will be a complete car wreck.

    Link to this comment Reply
  29. Rich

    Not sure about the analogy?
    Schools not only need to predict - they actually have to make the weather on that certain day in August. Predicting temperatures in degrees C? How about using a brand new and untested temperature scale? Over familiar Britain? No the new topography in English and Maths has many more mountains in it. How about if the air mass was measured 6 years ago in a different place, by different people, using another different scale and the weather we make has to match with that distant forecast?
    Outstanding results are produced by a forensic analysis and understanding of where each child is in each of their subjects as they progress through the course. Key to this is accurate assessment data (raw mock marks) and some indication of the grades these might achieve. This is how it works. Schools can't be blamed for using every tool available to try to estimate what grade Jack's 62% in his mock will turn into in the Summer exams. Cautious we are - very. In need of over simplistic patronising analogies?

    Link to this comment Reply
  30. Christine Watson

    In all of these comments, no mention of the 5% that will be awarded grade U in 2017 in maths using the comparability model. Yes, over time the NRT will allow adjustments to the percentage at the anchor points but these are not designed to measure the proportion at U. Is the intention that 'raising standards' means the boards will maintain this 5%? Note that it is a far smaller proportion for English GCSE.

    Link to this comment Reply
  31. Holly

    Hey. I wanted to ask that lets say the gt for 2016 was 70 points out of 100 to get an A*, then in 2017, would the gt be from 65-75 points, or could it go beyond higher. What im asking is that for the 2017 exams will the gt change by a huge amount or remain relatively the same, (like max. increase be of 5 points.)could a gt shoot from 70 points in 2016 to 90 points in 2017, i mean, would candidates improve by that extent, really?
    Thankyou. 🙂

    Link to this comment Reply
  32. George Martin

    The craziest thing here is that, while I totally inderstand that it is impossible to predict grades, the exam boards are STIll asking us to do so!!

    Link to this comment Reply
    • Cath Jadhav

      We have checked with the exam boards offering GCSEs and A levels in England and they no longer collect estimated/forecast grades.

      Link to this comment Reply
  33. Julie Clark

    I am a parent of twins sitting the new gcse maths paper. One of my Daughter's has been studying the foundation level for two year's and my other Daughter has been in the higher level class. Yesterday school called me to say that they would like my Daughter who has studied the higher level to now sit the foundation paper instead (along with several other pupils) as they are worried that they will not pass the higher paper. Wanting what's best for her I agreed but I am now extremely worried for my other Daughter because if all the higher level pupils sit the foundation exam then they will inevitably push up the grade boundaries causing more foundation level pupils to fail. From what I gather this is happening in schools all over the country, how will you ensure that all the current foundation level pupils will not be affected and still achieve a pass?

    Link to this comment Reply
    • Cath Jadhav

      Schools are responsible for making decisions about which tier to enter each student for. We have been working with exam boards to make sure that standards are in line across foundation and higher tier. We published more information about this in one of our other blogs, choosing the right tier.

      Link to this comment Reply
  34. Brian Lockwood

    Whose idea was it to have grade 9 High anyway?

    Link to this comment Reply

Leave a comment