Validating an Approach to Assess Sensor Perception Reliabilities without Ground Truth

A reliable environment perception is a requirement for safe automated driving. For evaluating and demonstrating the reliability of the vehicle's environment perception, field tests offer testing conditions that come closest to the vehicle's driving environment. However, establishing a reference ground truth in field tests is time-consuming. This motivates the development of a procedure for learning the vehicle's perception reliability from fleet data without the need for a ground truth, which would allow learning the perception reliability from fleet data. In Berk et al. (2019), a method based on Bayesian inference to determine the perception reliability of individual sensors without the need for a ground truth was proposed. The model utilizes the redundancy of sensors to learn the sensor's perception reliability. The method was tested with simulated data. In this contribution, we further explore and validate the method by utilizing real data, including ground truth data based on high-resolution LIDAR and human labeling. An area with overlapping field of view from five sensors is selected for the analysis. A basic association method is used to compare the object data obtained from the different sensors. Finally, we compare the sensor perception reliabilities learned from the Bayesian inference model with the sensor perception reliabilities determined from the labeled ground truth. In this paper, it is shown that the model introduced in Berk et al. (2019) can approximate the reference data based on the provided ground truth. The estimated parameters of the model do not perfectly correspond to the sensor reliabilities but are of the same order of magnitude as when derived from the ground truth.