Robust Depth Estimation in Foggy Environments Combining RGB Images and mmWave Radar

In this paper, we propose a robust depth estimation strategy that uses RGB images and mmWave radar data to deal with limited visibility in foggy environments. While the state-of-the-art RGB or LiDAR-based depth estimation works well in scenarios with good visibility, their performance dramatically degrades in the presence of fog. In contrast, mmWave radar sensors are not affected by fog and hence are a promising complement. To leverage this property of mmWave radar, we combine RGB image-based depth estimation with radar information. The proposed combination is an extension of the Sparse-to-Dense (S2D) model. Moreover, a weight-based sensor fusion strategy is presented to improve system performance. Our experiments show that a fog density of meteorological optical range (MOR) less than 50m leads to strongly degraded performance for RGB image-based and LiDAR-based depth estimation. For a MOR of 30m in our dataset, the experiments show an improvement of 26% in mean square error for our proposed approach compared to the combination of RGB images and LiDAR data.