When are we guessing? An investigation of the impact of guessing on the validity of results in the assessment of students in large-scale assessment programs

The 21st century shift towards economic rationalism has included a trend towards measuring the outcomes of education and, equipped with these data, placing a higher priority on accountability in all sectors of the educational community. Corresponding to this shift, policy makers and government officials continue to demand better quality information on which to make data-informed decisions and allocate resources. To measure educational outcomes, for instance, stakeholders routinely look to national and international large-scale testing programs for indicators of success in achieving national goals and to identify areas for systemic attention. A majority of these large-scale assessments use multiple-choice item types to assess student ability for the reporting of the outcomes of student achievement against curriculums and national or international standards. Students are encouraged to attempt all questions in these assessments. By doing so it is likely that students will guess the answers to those items where the concepts assessed are beyond their capabilities and be rewarded with an incremental increase in their results. This has the potential to threaten the validity of students’ overall results, due to the chance of correctly guessed responses inflating assessment results beyond students’ actual ability levels. This PhD research investigated the potential impact of – and proposed a statistical solution for – guessing in large-scale education assessments. Specifically, it examined the outcomes from a novel application of the Rasch analysis technique, which quantified and adjusted for the measurement error associated with guessing and, consequently, increased the validity of the outcomes of the assessments.