Metamorphic Testing: Testing the Untestable

Authors

Sergio Segura, University of Seville
Dave Towey, University of Nottingham Ningbo
Zhi Q. Zhou, University of WollongongFollow
Tsong Yueh Chen, Swinburne University of TechnologyFollow

RIS ID

132401

Publication Details

Segura, S., Towey, D., Zhou, Z. Quan. & Chen, T. Yueh. (2019). Metamorphic Testing: Testing the Untestable. IEEE Software, Online First 1-12.

Abstract

What if we could know that a program is buggy, even if we could not tell whether or not its observed output is correct? This is one of the key strengths of metamorphic testing, a technique where failures are not revealed by checking an individual concrete output, but by checking the relations among the inputs and outputs of multiple executions of the program under test. Two decades after its introduction, metamorphic testing has become a fully-fledged testing technique with successful applications in multiple domains, including online search engines, autonomous machinery, compilers, Web APIs, and deep learning programs, among others. This article serves as a hands-on entry point for newcomers to metamorphic testing, describing examples, possible applications, and current limitations, providing readers with the basics for the application of the technique in their own projects.