Test Method for Measuring the Simulation-to-Reality Gap of Camera-based Object Detection Algorithms for Autonomous Driving

The validation of automated driving requires billions of kilometers of test drives to be performed so that safety-in-use is assured. It is difficult to validate it only making use of data acquired on field tests in public roads due to the lack of controllability, e.g. over environment conditions. Therefore, the automotive industry relies on test drives to be executed on a proving ground under controlled conditions or in environment simulation software. The first is realistic, but costly in terms of time and effort. The latter provides a high level of reproducibility, but it is still uncertain how valid the delivered test results are. In this paper, a test method for measuring the simulation-to-reality gap is proposed. For this purpose, a test scenario is defined, built on a proving ground and reproduced in two environment simulation software. Four different environment conditions are considered: day, night, fog and rain. The video data of the real and simulated test drives are recorded and fed into a series-produced multi-class object detection algorithm for automated driving. Performance metrics are calculated across the real and virtual domains. Finally, the test results are compared so that the simulation-to-reality gap concerning object detection is measured.