Open-Set Object Detection for the Identification and Localization of Dissimilar Novel Classes by means of Infrastructure Sensors

This research focuses on solving challenges related to identifying unfamiliar object categories in the realm of Open-Set Object Detection (OSOD) using infrastructure sensors. Traditional camera-based OSOD systems struggle to generate proposals for dissimilar novel classes due to a lack of feature similarity. This research introduces a novel approach named Fusion Object Detector (FOD), which emphasizes the localization and identification of semantically dissimilar unknown objects through a multimodal fusion architecture involving infrastructure-mounted cameras and LiDARs. FOD leverages a camera-based closed-set object detector for the identification of known class objects, while simultaneously utilizing clusters derived from fused LiDAR point clouds for the detection of unknown class objects. This research work also presents a novel dataset named Thermal camera and LiDAR in Infrastructure Dataset (TLID). TLID comprises fused sensor measurements from multiple thermal cameras and LiDARs mounted in three urban crossings of Ingolstadt city and at CARISSMA outdoor test track. The proposed methodology is evaluated using both an in-house dataset and a publicly available infrastructure dataset for the task of OSOD. The results quantify the importance of multimodal sensor information for the task of identifying dissimilar unknown objects.