During the first half of this year we carried out research to evaluate the expected difficulty of exam boards’ sample GCSE maths papers developed as part of the current reforms and their approach to the assessment of mathematical problem solving. When we released those results we said that we would consider the benefits of undertaking similar research for other new GCSEs or A levels prior to exam board specifications being accredited.
Given the amount of change in GCSE science we decided that it would be appropriate to undertake some supplementary research looking at the various sample papers (for biology, chemistry and physics) that have been submitted to us as part of accreditation. This work will run parallel to accreditation and will inform our accreditation decisions.
So what are we doing? Well, we’re doing two things.
First, we are performing a strand of research similar to that which we used when looking at the GCSE maths papers. We’ve asked just over 100 science teachers to judge the expected difficulty of more than 6,000 questions taken from the exam boards’ sample papers and the question papers they used in 2014. Each teacher was been asked to compare pairs of questions and decide which of them their students would find more difficulty to answer. They’ve been asked to do this for 1,000 pairs of questions and the questions and teachers have been split by subject – biology, chemistry and physics – to make meaningful comparisons. The teachers started judging items on 12 August and they concluded today (24 August). We will now analyse the judgements made to provide evidence to the accreditation panel about the expected difficulty of the sample papers.
This piece of work is quite innovative as the methodology has not been widely used to evaluate science questions – the type of questions may influence the effectiveness of the approach – and some of the statistical analyses we will be performing are also new to this area.
As this hasn’t been done before we are uncertain about what we will find and how meaningful the data will be so we will be interpreting the results carefully. To make sure we do have information we can use to support accreditation we’re not stopping there...
We will also be using quantitative evidence from the assessments used in the current version of the GCSE science specifications. We will be providing our accreditation panel with detailed data about the effectiveness of the current assessments to provide a context for their consideration of the sample papers. This will support the panel when coming to a view about whether or not the proposed sample question papers are likely to be of the right level of difficulty – sufficiently difficult to reflect the new content, but not so difficult that they fail to effectively differentiate between students of different abilities.
Each specification will still need to comply with all of our published regulatory requirements related to GCSEs in order to be accredited under our processes, but this work will support the panel in judging compliance.
Dr Ian Stockford
Executive Director for General Qualifications, Ofqual
2 comments
Comment by Helen posted on
Very interesting and stimulating to be part of such an inovative approach. Thank you for the opportunity to be part of this.
Like the DNA style model.
It is only possible to find out new things by trying lots of things and throwing out what doesn't work and keeping what does. I will be most intrested to see how this progresses.
Wishing you all the best
Helen
Comment by Chris Colclough posted on
I am very surprised that there are not many tried and tested ways of checking the level of difficulty of examination questions. Those used in higher education for example. Have academics not been consulted over this issue?
You do seem to be rather reinventing the wheel.
However as a biologist, I do like the DNA model. It does rather over complicate the process though.