In-Vehicle Object-Level 3D Reconstruction of Traffic Scenes

Emerging automotive applications such as in-vehicle Augmented Reality (AR) and fully automated parking require a comprehensive understanding of the vehicle's three-dimensional surrounding represented as an object-level environmental model. In this model, not only 3D poses (positions and orientations) and 3D sizes of detected objects are registered, but 3D shapes (geometries) need to be reconstructed precisely. A combination of 3D object detection and 3D surface reconstruction techniques, referred to as object-level 3D reconstruction, is fundamental to building such environmental models. However, the possibilities to incorporate object-level 3D reconstruction in a car have not been sufficiently explored either in academic research or in the industry. This primarily stems from the cost and resource constraints associated with the automotive domain. In this paper, we address these constraints by proposing implementations of in-vehicle object-level 3D reconstruction in two specific use cases: (i) augmented reality and (ii) automated parking. For augmented reality, we propose a cost-efficient solution called monocular 3D Shaping that requires only a single frame from a monocular camera as input. For automated parking, we propose a resource-efficient alternative that generates more precise 3D reconstruction results by taking advantage of additional 3D sensors (such as Lidars). The crux of our proposed approaches lies in the use of a Latent Shape Space, where various 3D shapes are represented using only two parameters. As a result, highly complex 3D shapes can now be transmitted using a low-to medium-bandwidth in-vehicle communication infrastructure in a cost-effective manner.