Non-deterministic model validation methodology for simulation-based safety assessment of automated vehicles

Safeguarding and type approval of automated vehicles is a key enabler for their market launch in our complex traffic environment. Scenario-based testing by means of computer simulation is becoming increasingly important to cope with the enormous complexity and effort. However, there is a huge gap when assessing the safety of the virtual vehicle while the real vehicle will drive on the road. Simulation must be accompanied by model validation to ensure its credibility since errors and uncertainties are inherent in every model. Unfortunately, this is rarely addressed in the current literature. In this paper, a modular process is presented covering both model validation and safeguarding. It is characterized by the fact that it quantifies a large number of errors and uncertainties, represents them in the form of an error model, and ultimately integrates them into the safeguarding results. It is applied to a type-approval regulation for the lane-keeping behavior of a vehicle under various scenario conditions. The paper contains a thorough validation of the methodology itself by comparing its results with actual ground truth values. For this comparison, a binary classifier and confusion matrices are used that relate the binary type-approval decisions. The classifier demonstrates that the methodology of this paper identifies a systematic error of the simulation model across several safeguarding scenarios. Finally, the paper provides recommendations for alternative configurations of the modular methodology depending on different requirements.