Quantifying the LiDAR Sim-to-Real Domain Shift: A Detailed Investigation Using Object Detectors and Analyzing Point Clouds at Target-Level

Sebastian Huch, Luca Scalerandi, Esteban Rivera, Markus Lienkamp

Research output: Contribution to journalArticlepeer-review

10 Scopus citations

Abstract

LiDAR object detection algorithms based on neural networks for autonomous driving require large amounts of data for training, validation, and testing. As real-world data collection and labeling are time-consuming and expensive, simulation-based synthetic data generation is a viable alternative. However, using simulated data for the training of neural networks leads to a domain shift of training and testing data due to differences in scenes, scenarios, and distributions. In this work, we quantify the sim-to-real domain shift by means of LiDAR object detectors trained with a new scenario-identical real-world and simulated dataset. In addition, we answer the questions of how well the simulated data resembles the real-world data and how well object detectors trained on simulated data perform on real-world data. Further, we analyze point clouds at the target-level by comparing real-world and simulated point clouds within the 3D bounding boxes of the targets. Our experiments show that a significant sim-to-real domain shift exists even for our scenario-identical datasets. This domain shift amounts to an average precision reduction of around 14% for object detectors trained with simulated data. Additional experiments reveal that this domain shift can be lowered by introducing a simple noise model in simulation. We further show that a simple downsampling method to model real-world physics does not influence the performance of the object detectors.

Original languageEnglish
Pages (from-to)2970-2982
Number of pages13
JournalIEEE Transactions on Intelligent Vehicles
Volume8
Issue number4
DOIs
StatePublished - 1 Apr 2023

Keywords

  • Autonomous vehicles
  • LiDAR
  • deep learning
  • domain shift
  • object detection
  • point cloud
  • simulation
  • synthetic data

Fingerprint

Dive into the research topics of 'Quantifying the LiDAR Sim-to-Real Domain Shift: A Detailed Investigation Using Object Detectors and Analyzing Point Clouds at Target-Level'. Together they form a unique fingerprint.

Cite this