Data-Driven Assessment of Parameterized Scenarios for Autonomous Vehicles

Highly automated and autonomous driving systems are usually tested for their safe behavior using a so-called scenario-based testing approach. A common practice is to let experts create parameterized scenarios by selecting and varying parameters of a given scenario type, e.g., the initial speed of the participating vehicles. By assigning concrete values to the selected parameters, scenario instances are generated, which may be used as test scenarios for the driving system under test (SUT). For the generation of test cases, parameterized scenarios typically serve as input. Most works assume parameterized scenarios to be given without evaluating their quality. However, a parameterized scenario may be insufficient, leading to inadequately and incomplete generated test cases, unreliable test results, and even incorrect conclusions about the safety of the SUT. As contribution of this work, we present a quality criterion and a novel data-driven assurance approach to assess parameterized scenarios. We consider the quality of a parameterized scenario to be acceptable if it contains at least all scenario instances collected in real traffic for the studied scenario type. For this containment check, search-based techniques are used. We show experiments for a parameterized lane change scenario using 6736 lane change recordings from real traffic for the assessment. The experiment results show that in addition to shortcomings of a parameterized scenario, those of the simulation setup can be revealed.