The multi-semiotic expression of emotion in storytelling performances of Cinderella: A focus on verbal, vocal and facial resources

It is widely recognised that emotion in spoken communication is conveyed multimodally; not only through wordings but also through a diverse range of other communicative resources such as vocal features, gesture, facial expression and posture (Abercrombie, 1968; Burns & Beier, 1973; Mehrabian, 1972; Ngo et al., 2022; Scherer & Ellgring, 2007; Scherer et al., 1984; Wallbott, 1998). Effective multimodal expression and interpretation of emotion in spoken English can be a challenge, especially when speakers of English as an additional language (EAL) are involved (Brown, 1977/1990; Dewaele & Moxsom-Turnbull, 2019; Lorette & Dewaele, 2019; Rintell, 1984; Roach, 2009; Unsworth & Mills, 2020). This study thus aims to explore the division of labour and interplay of multi-stratal and multi-semiotic meaning across verbal, vocal and facial resources in the expression of emotion in spoken English.

The study deploys a Systemic Functional Semiotic (SFS) perspective (e.g., Halliday, 1978; Halliday & Matthiessen, 2014; Jewitt et al., 2016; Kress, 2010; Matthiessen, 2007, 2009; Ngo et al., 2022; van Leeuwen, 1999) to explore and establish how wordings, vocalisations and facial expressions construe emotion, both separately and synergistically, in eight storytelling performances of Cinderella. While emphasis is placed on the definition, analysis and description of the vocal qualities deployed to express emotion and in developing a clear framework and consequent analytical approach, the study also focuses on how verbal, vocal and facial semiotic resources combine to create multi-semiotic messages that affectively colour the phases and stages of the storytelling performances. The results of this exploration are consolidated in an exploratory system network of SEMOGENIC VOCAL QUALITIES which bundle together to realise affectual meanings in spoken English. A key argument in this thesis is that affectual vocalisation interacts with verbiage and facial expression as an ensemble of resources that integrate in the discourse semantic system of AFFECT.

The significance of this study is both theoretical and practical. Theoretically, it constitutes a potential contribution to the development of a systemic functional multi-stratal and multi-semiotic account of the phonological resources that inscribe affectual meaning in spoken language. Practically, it offers a foundation for developing an EAL multi-semiotic literacy toolkit to explore and describe the grammar of emotion in the context of storytelling performances. Such a recontextualisation of the multi-stratal and multi-semiotic frame proposed in this thesis could hold great value for second language teaching contexts. The results of this project could inform the practices of EAL teacher training courses, offering well-founded principles to support students’ interpretation and use of the multi-semiotic expression of emotions. In addition, a SFS analytical toolkit of this kind could advance work in and beyond the EAL pedagogical context of this study, such as in higher education interaction more generally (e.g., Hood, 2010, 2011; Hood & Hao, 2021; Macnaught, 2018; Martin & Dreyfus, 2015), in clinical interaction in healthcare (Butt et al., 2012; Henderson-Brooks, 2006; Moore et al., 2010) and in automated emotion detection as a tool for reception studies of multimodal ensembles in literary text reading (e.g., Montaña & Alías, 2016; Wegener et al., 2017), among others.