Adversarial semantic scene completion from a single depth image

Yida Wang, David Joseph Tan, Nassir Navab, Federico Tombari

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

26 Scopus citations

Abstract

We propose a method to reconstruct, complete and semantically label a 3D scene from a single input depth image. We improve the accuracy of the regressed semantic 3D maps by a novel architecture based on adversarial learning. In particular, we suggest using multiple adversarial loss terms that not only enforce realistic outputs with respect to the ground truth, but also an effective embedding of the internal features. This is done by correlating the latent features of the encoder working on partial 2.5D data with the latent features extracted from a variational 3D auto-encoder trained to reconstruct the complete semantic scene. In addition, differently from other approaches that operate entirely through 3D convolutions, at test time we retain the original 2.5D structure of the input during downsampling to improve the effectiveness of the internal representation of our model. We test our approach on the main benchmark datasets for semantic scene completion to qualitatively and quantitatively assess the effectiveness of our proposal.

Original languageEnglish
Title of host publicationProceedings - 2018 International Conference on 3D Vision, 3DV 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages426-434
Number of pages9
ISBN (Electronic)9781538684252
DOIs
StatePublished - 12 Oct 2018
Event6th International Conference on 3D Vision, 3DV 2018 - Verona, Italy
Duration: 5 Sep 20188 Sep 2018

Publication series

NameProceedings - 2018 International Conference on 3D Vision, 3DV 2018

Conference

Conference6th International Conference on 3D Vision, 3DV 2018
Country/TerritoryItaly
CityVerona
Period5/09/188/09/18

Keywords

  • Adversarial training
  • Depth image
  • Latent space
  • Scene completion

Fingerprint

Dive into the research topics of 'Adversarial semantic scene completion from a single depth image'. Together they form a unique fingerprint.

Cite this