Metamorphic Robustness Testing of Google Translate

Authors

DIckson Lee
Zhi Q. Zhou, University of WollongongFollow
T.H. Tse, University Of Hong Kong

RIS ID

146058

Publication Details

Lee, D., Zhou, Z. & Tse, T. (2020). Metamorphic Robustness Testing of Google Translate. Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020 (pp. 388-395). New York, United States: Association for Computing Machinery.

Abstract

© 2020 ACM. Current research on the testing of machine translation software mainly focuses on functional correctness for valid, well-formed inputs. By contrast, robustness testing, which involves the ability of the software to handle erroneous or unanticipated inputs, is often overlooked. In this paper, we propose to address this important shortcoming. Using the metamorphic robustness testing approach, we compare the translations of original inputs with those of follow-up inputs having different categories of minor typos. Our empirical results reveal a lack of robustness in Google Translate, thereby opening a new research direction for the quality assurance of neural machine translators.