Purpose - The purpose of this paper is to address the issue of failing to psychometrically test questionnaire instruments when measuring university students' attitudes towards plagiarism. These issues are highlighted by a psychometric evaluation of a commonly used (but previously untested) plagiarism attitudinal scale. Design/methodology/approach - The importance of psychometric testing is shown through an analysis of a commonly used scale using modern techniques (e.g. Rasch analysis) on 131 undergraduate education students at an Australian university. Findings - Psychometric analysis revealed the scale to be unreliable in its present form. However, when reduced to an eight-item subscale it became marginally reliable. Research limitations/implications - The main implication of this paper is that questionnaire instruments cannot be assumed to function as they are intended without thorough psychometric testing. Practical implications - The paper offers valuable insight into the psychometric properties of a previously untested but commonly used plagiarism attitudinal scale. Originality/value - The paper offers a straightforward and easy to understand introduction to researchers in higher education who use questionnaires/surveys in their research but lack an understanding of why psychometric testing is so critical. While similar papers have been written in other fields which advocate psychometric approaches, such as Rasch analysis, this has not been the case in higher educational research (or mainstream educational research for that matter).