Doctor of Education
Faculty of Social Sciences
Sitepu, Gantina, Enhancing rater reliability in the assessment of academic writing: An SFL analysis of raters’ rating and perceptions, Doctor of Education thesis, Faculty of Social Sciences, University of Wollongong, 2014. http://ro.uow.edu.au/theses/4433
This study investigated and aimed to enhance teachers’ assessment practices related to academic writing in a language school in Indonesia. For this purpose, the more explicit and linguistically informed marking criteria were developed and introduced to the teachers’ involved in teaching English for Academic Purposes (EAP) classes in the given language school through an intervention. The study adopted an actionoriented case study to capture the teachers’ marking experiences, which comprise their accounts of issues of rubrics and marking consistency, and their actual performance in the assessment of argument essays written by 20 EAP students. The study drew on Systemic Functional Linguistics (SFL) and Appraisal theory (Halliday 1978; 1985a, Halliday and Hasan 1985, Halliday and Matthiessen 2004, Martin 1984, Martin and White 2005; 2007) to analyse teachers’ accounts and inform the design of the new assessment criteria.
The main purpose of the study is to identify the effect of an SFL informed rubric through training on marking consistency to enhance rater reliability. It aims to design an SFL informed model based on essay analysis and implementing this through the training of teachers. In order to achieve this, specifically, the thesis aims to investigate teachers’ pre-perceptions about marking using an existing marking rubric (pre-intervention) and design a new marking model using an SFL framework of genre, Field, Tenor and Mode, and trains the teachers (intervention). Finally, it aims to identify the changes from teachers’ pre-perceptions about existing rubrics to their post perceptions about the new rubrics and marking practices.
This study employed a qualitative approach, drawing on six EAP teachers’ marking experiences.
The research design comprised three main parts that addressed the research questions. These involved: (i) investigating teachers’ perceptions and sources of problems in marking, (ii) designing and implementing the linguistically informed marking criteria, and (iii) investigating teachers’ opinions and actual marking practices (pre- and post-intervention).
This study had three major findings. The first was that linguistically informed assessment criteria, which became shared knowledge of teachers, had a key role in enhancing the quality of assessment in academic writing. The improved quality of teachers’ assessment was indicated by the higher consistency rates among the teachers in their marking of student essays when using the SFL-informed rubric. The second finding was that teachers response was significantly more positive when talking about their marking experience using the SFL informed rubric than when using the existing rubric. Finally, teachers were well aware of their teaching, marking practices and students’ needs, and they recognised the gaps between the three.
The contributions of the study were: (i) it made visible what counts in successful student essays, (ii) it expanded the application of SFL theory by trying it out on the assessment of academic writing in a non-English speaking country, and (iii) it stimulated a discussion on the relationship between teachers’ perceptions and marking practices.