IMAGE-GUIDED NEURAL OBJECT RENDERING

Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

Publikation: KonferenzbeitragPapierBegutachtung

24 Zitate (Scopus)

Abstract

We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis. The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours & sightseeing, the digital inspection of historical artifacts). A core component of our work is the handling of view-dependent effects. Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object. As input data we are using an RGB video of the object. This video is used to reconstruct a proxy geometry of the object via multi-view stereo. Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering. This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts. To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects. Based on these estimations, we are able to convert observed images to diffuse images. These diffuse images can be projected into other views. In the target view, our pipeline reinserts the new view-dependent effects. To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results. Using this image-guided approach, the network does not have to allocate capacity on “remembering” object appearance, instead it learns how to combine the appearance of captured images. We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.

OriginalspracheEnglisch
PublikationsstatusVeröffentlicht - 2020
Veranstaltung8th International Conference on Learning Representations, ICLR 2020 - Addis Ababa, Äthiopien
Dauer: 30 Apr. 2020 → …

Konferenz

Konferenz8th International Conference on Learning Representations, ICLR 2020
Land/GebietÄthiopien
OrtAddis Ababa
Zeitraum30/04/20 → …

Fingerprint

Untersuchen Sie die Forschungsthemen von „IMAGE-GUIDED NEURAL OBJECT RENDERING“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren