Testing and Understanding the Robustness of Autonomous Driving Software

Autonomous driving (AD) has been one of the most attractive applications in the era of artificial intelligence (AI). Scenario-based simulation testing is a major approach to testing the software components of AD vehicles. In general, there are two approaches to testing software systems (including AD systems), namely, positive testing and negative testing (i.e. robustness testing). Positive testing checks the consistency between the system’s specifications and implementation, aiming to ensure that the system under test (SUT) performs correctly for expected inputs. By contrast, robustness testing attempts to ensure that the SUT does not behave badly when the input is invalid or unexpected. In the context of practical AD testing, especially for complex or very random scenarios, it is often difficult to determine whether robustness tests pass or fail. This is a typical situation known as the oracle problem. Metamorphic testing (MT) has been proven effective in alleviating the oracle problem. Instead of checking the correctness of each individual test result, MT examines the relations among multiple test case executions. These relations are called metamorphic relations (MRs) — necessary properties of the software under test (SUT). Violations of MRs reveal the presence of bugs. In this thesis, we adopt MT and propose a metamorphic fuzz testing (MFT) approach, which enables testers to use fuzzing to proactively generate scenarios that are more complex and random than those produced in the existing literature, and then distinguish genuine failures from false alarms generated during the fuzzing process. Compared with the existing literature, MFT can generate completely different types of failure-causing scenarios, and therefore reveal previously unknown bugs and robustness issues. It is to be noted that, during fuzz testing, when the AD system behaves differently from the tester’s expectation, it may not necessarily mean that the SUT is erroneous, because the SUT could have been designed in such a complex way that the tester might not fully understand it. We thus use the concept of metamorphic exploration (ME) — a useroriented testing and system understanding approach that makes use of hypothesised MRs — in combination with fuzzing and statistical analysis to enable better understanding of the AD system’s behaviour in extreme traffic situations. Finally, this thesis extends the approach of MR generation from using traffic flows to using constructed maps.