Year

2024

Degree Name

Doctor of Philosophy

Department

School of Computing and Information Technology

Abstract

Advanced Driver Assistance Systems (ADASs) and Autonomous Vehicles (AVs) are becoming increasingly popular. These technologies have the potential to revolutionise transportation, but they also face a number of challenges in terms of safety. Simulation testing —– the use of simulations for testing –— is essential when developing ADAS and autonomous driving (AD) systems. It plays a vital role in augmenting and speeding up real-world testing. It is reproducible, scalable and reduces development costs. However, a challenge for simulation testing can occur when determining whether or not the SUT or the simulator correctly produces the intended behaviors. The effectiveness of simulation testing depends on the simulation platform’s quality: if the simulation platform is unreliable, the simulation findings could be inaccurate and, therefore, useless. The oracle problem is a circumstance where it can be too difficult (or expensive) to verify or validate the simulation platforms due to a lack of appropriate test oracles or ways to establish the validity of the program output or behavior. The use of testing standards and protocols is essential for designing and developing autonomous systems. However, a lack of common testing mechanisms and strategies makes it challenging for testers and industries to achieve their targets.

One of the popular methods to alleviate the oracle problem is Metamorphic Testing. When applying MT, the identification of Metamorphic Relations is critical. This thesis aims to provide a simulation-based-MT approach by identifying a series of MRs to implement testing standards and protocols. The empirical studies were conducted on the Verification and Validation (V&V) of three popular driving simulators for ADAS development (MATLAB & Simulink, 51Sim-one Cloud, and esmini) and examining their embedded ADAS systems.

This research conducts case studies using existing ADAS testing standards, such as the European New Car Assessment Programme (Euro NCAP), and major simulation testing platforms including MATLAB Simulink, 51Sim-One, and Esmini. These case studies find that combining existing testing standards with MRs can not only alleviate the oracle problem of the former but also help to detect previously unknown software issues in both the simulation platforms and the ADAS software. The software vendors have confirmed our reported issues.

This thesis also relates the concept of MT to ISO standards including ISO 34502 (Road vehicles—Test scenarios for automated driving systems—Scenario based safety evaluation framework) and ISO 21448 (Road vehicles—Safety of the intended functionality).

In the software engineering literature, although MT has already been applied to the testing of simulation software and AD systems, to the best of our knowledge, this is the first research that investigates specifically AD simulation software as well as the AD systems.

FoR codes (2008)

080306 Open Software, 080309 Software Engineering, 080110 Simulation and Modelling, 080111 Virtual Reality and Related Simulation

Download

Share

COinS
 

Unless otherwise indicated, the views expressed in this thesis are those of the author and do not necessarily represent the views of the University of Wollongong.