Doctor of Philosophy
Department of Psychology - Faculty of Health & Behavioural Sciences
Smith, Brooke L, Conventional versus computer-based administration of measures of cognitive ability: an analysis of psychometric, behavioural, experiential and relativity of equivalence, PhD thesis, Department of Psychology, University of Wollongong, 2003. http://ro.uow.edu.au/theses/165
Test equivalence can be evaluated in terms of four aspects: psychometric, behavioural, experiential and individual differences (i.e., relativity of equivalence) (Honaker, 1988). This thesis comprises two studies designed to explore the equivalency of scores obtained on conventional versus computerised measures of cognitive ability in terms of all four criteria. In addition, this thesis provides theoretical clarification of the link between computer anxiety and performance. This objective was achieved by developing a theoretical model for exploring the effect of computer anxiety on computer based test (CBT) performance and test equivalence. The first study was conducted in an organisational setting and employed a repeated-measures mixed design to examine the equivalency of two Australian Defence Force (ADF) personnel selection tests. Hence, the first study was divided into two phases. Phase 1 of Study 1 examined the equivalence of Test MX, a multiple-choice test of simple mathematics, using data from a sample of 685 ADF applicants. Phase 2 explored the equivalence of Test C, a timed clerical aptitude test, based on test score information from 781 ADF applicants. Study 2, on the other hand, was conducted in an educational setting and employed 180 undergraduate students to examine the equivalency of the Australian Council for Education Research Quantitative Test (ACER-AQ). Psychometrically, the results of Study 1 showed that scores obtained on the conventional and computerised versions of Test MX and Test C were not equivalent. Although similarities in rankings were found across the computer and conventional tests, the results demonstrated that the two formats differed in terms of mean scale scores, dispersion, and distribution of scores. From a behavioural perspective, differential speededness and/or omit patterns were found across the conventional and computerised versions of Test MX and Test C. Moreover, the results of Study 1 suggest that the two administration modes were not experientially equivalent. However, these experiential differences did not affect computer-conventional equivalency. With respect to relativity of equivalence, the results of Study 1 indicated that scores from the two formats did not vary as a function of the individual characteristics of interest (i.e., test anxiety, trait anxiety, state anxiety, computer anxiety, computer experience, computer thoughts). In extending theory and research on computer anxiety, the results of Study 1 provided some support for the theoretical model proposed by demonstrating that computer thoughts mediated the main effect of computer anxiety on CBT performance. In contrast, Study 2 employed a four-group, counterbalanced, repeated-measures design to examine the equivalency of the ACER-AQ in terms of all four equivalency criteria. The results of Study 2 demonstrated that the conventional and computerised formats of the ACER-AQ were equivalent in terms of psychometric and individual difference aspects. However, the two forms differed in terms of behavioural and experiential aspects. Taken together, the present findings suggest that test-taking behaviour and experiential factors need to be examined as potential sources of construct-irrelevant variance in computer-based testing. Practical and empirical implications for computer-based assessment are discussed. In addition, future research opportunities and theoretical developments pertaining to the computer anxiety-performance linkage are considered.