Metamorphic Fuzz Testing of Autonomous Vehicles

Authors

Jia Han, University of Wollongong
Zhi Q. Zhou, University of WollongongFollow

RIS ID

146056

Publication Details

Han, J. & Zhou, Z. (2020). Metamorphic Fuzz Testing of Autonomous Vehicles. Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020 (pp. 380-385). New York, United States: Association for Computing Machinery.

Abstract

© 2020 ACM. Driving simulation is the primary approach for testing the software components of autonomous vehicles. This paper presents an automated testing method, termed metamorphic fuzz testing (MFT), in the context of simulation testing of autonomous driving. MFT differs from existing fuzzing techniques in the following two stages: First, it can generate "unrealistic"scenarios where scenes of the virtual world are refreshed frequently (so obstacles can suddenly appear / disappear) - -this is to test the self-driving vehicle's robustness in the face of unexpected situations. In the second stage, MFT uses metamorphic relations as a filtering or debugging tool to distinguish between genuine failures and false alarms yielded in the first stage. We conduct empirical studies using the real-life Baidu Apollo self-driving system, recording a genuine failure rate of 3.7%. We have reported some of the detected failures to the Apollo team and received their confirmation. Our testing method is platform-independent and, therefore, can be applied to other autonomous driving systems and advanced driver-assistance systems (ADAS).