Did We Test All Scenarios for Automated and Autonomous Driving Systems?

To ensure safety and functional correctness of automated and autonomous driving systems, virtual scenario-based testing is used. Experts derive traffic scenario types and generate instances of these types with the support of test generation tools. Since driving systems operate in a real-world environment, it is always possible to find a new scenario type as well as new instances of scenario types that are different from all other scenario types and instances. Thus, the testing process to find faulty behavior may continue forever. There is a practical need for test ending criteria for both of the following problems: Did we test all scenario types? Did we sufficiently test each type with specific instances? We address the first question and present a suitable test ending criterion and methodology. Whether the system is tested in each scenario type is reduced to the question whether all test scenarios are known. We analyze driving data to provide a statistical guarantee that all scenario types are covered. We model this as a Coupon Collector's Problem. We present experimental results for the application of this model to different driving tasks of automated and autonomous driving systems.