Redundant Sensor-Based Perception Sensor Reliability Estimation from Field Tests without Reference Truth

The introduction of autonomous vehicles has gained significant attention due to its potential to revolutionize mobility and safety. A critical aspect underpinning the functionality of these autonomous vehicles is their sensor perception system. Demonstrating the reliability of the environment perception sensors and sensor fusion algorithms is, therefore, a necessary step in the development of automated vehicles. Field tests offer testing conditions that come closest to the environment of an automated vehicle in the future. However, a significant challenge in field tests is to obtain a reference truth of the surrounding environment. Here, we propose a pipeline to assess the sensor reliabilities without the need for a reference truth. The pipeline uses a model to estimate the reliability of redundant sensors. To do this, it relies on a binary representation of the surrounding area, which indicates either the presence or absence of an object. Therefore, the pipeline includes another step to convert object lists into this binary representation. Using the pipeline, we estimate the sensor reliabilities from object data derived from the Waymo dataset. Even though we are capable of obtaining close estimates of the sensor reliabilities we find out that the estimation of the sensor reliabilities is not robust for different parameter sets.