The use of vignettes to capture clinical psychology practicum competencies: vignette standardization and preliminary results
RIS ID
66394
Abstract
Converging evidence from several health disciplines suggests that field supervisor ratings of student competencies may be affected by systematic leniency and halo biases. Characteristics of the rating scales, rather than unskilled assessors, may be primarily responsible for these unsatisfactory assessment outcomes. The vignette method has yielded positive preliminary results. In the vignette approach, the supervisor is provided with pre-designed vignettes and asked to choose a vignette that best matches the student‘s performance. The current study is a multi-site initiative involving six universities in Australia and funded by an Australian Learning and Teaching Council (ALTC) grant. A key aim is to design, standardise, and evaluate a catalogue of vignettes to assess clinical psychology competencies in the field, and to compare outcomes from the vignette and rating-scale approaches. Preliminary results from the study are reported. The vignettes were designed following a rigorous process: (i) A group of experts drafted vignettes (V1) for each of 4 performance levels across 9 broad competency domains (36 vignettes), (ii) V1-vignettes were scrutinised within small group discussions or assigned to an expert in the psychology domain before revised V2-vignettes were produced, (iii) V2-vignettes were evaluated either within a small group of experts or by two blinded experts who independently revised the vignettes. Expert comments were considered before revised V3-vignettes were prepared for pilot testing, (iv) a web-based survey program was used to present the V3-vignettes to a group of experts (n=12) for evaluation and calibration. The experts reviewed each vignette independently, rated the vignettes in terms of their adequacy and efficacy, and calibrated each vignette using a visual analogue scale that ranged from 1 (unskilled) to 10 (competent), (v) The V3-vignettes were field tested by having a group of field supervisors (n=25) use the vignettes and the conventional rating scale to assess the competencies of psychology trainees at the end of placement. Field-test results found that the vignettes elicited a better distribution across competence levels than distributions previously elicited by rating scales. The majority of V3-vignettes received calibration scores that were satisfactory with regard to distribution across the full range. Vignettes that describe performance levels at either end of the competency continuum elicit calibration scores that suggest good agreement among experts, whereas vignettes depicting intermediate levels generally elicit calibration scores with moderate levels of agreement. Some V3-vignettes did not meet criteria for adequacy and require revision. Practicum competency ratings currently assigned to students by supervisors are likely to be much less reliable than is assumed, and merit urgent attention. Replacing the conventional rating scale with standardised vignettes is an innovation that has yielded good preliminary results and merits further investigation. Although the standardisation of vignettes is an arduous process, if successful, the technique has the potential to provide much more accurate data and appropriate feedback to students and faculty. The approach is also likely to be feasible for other disciplines.
Publication Details
Gonsalvez, C. J., Bushnell, J. A., Blackman, R., Deane, F. P., Bliokas, V. V., Nasstasia, Y., Nicholson-Perry, K., Allan, C. D., Knight, R., Shires, A. & Hyde, J. (2011). The use of vignettes to capture clinical psychology practicum competencies: vignette standardization and preliminary results. Proceedings of the Australian Technology Network Assessment Conference (pp. 31-31). Australia: ATN.