TriPlaneNet: An Encoder for EG3D Inversion

Ananta R. Bhattarai, Matthias Nießner, Artem Sevastopolsky

Publikation: Beitrag in Buch/Bericht/KonferenzbandKonferenzbeitragBegutachtung

2 Zitate (Scopus)

Abstract

Recent progress in NeRF-based GANs has introduced a number of approaches for high-resolution and high-fidelity generative modeling of human heads with a possibility for novel view rendering. At the same time, one must solve an inverse problem to be able to re-render or modify an existing image or video. Despite the success of universal optimization-based methods for 2D GAN inversion, those applied to 3D GANs may fail to extrapolate the result onto the novel view, whereas optimization-based 3D GAN in-version methods are time-consuming and can require at least several minutes per image. Fast encoder-based techniques, such as those developed for StyleGAN, may also be less appealing due to the lack of identity preservation. Our work introduces a fast technique that bridges the gap between the two approaches by directly utilizing the tri-plane representation presented for the EG3D generative model. In particular, we build upon a feed-forward convolutional encoder for the latent code and extend it with a fully-convolutional predictor of tri-plane numerical offsets. The renderings are similar in quality to the ones produced by optimization-based techniques and outperform the ones by encoder-based methods. As we empirically prove, this is a consequence of directly operating in the tri-plane space, not in the GAN parameter space, while making use of an encoder-based trainable approach. Finally, we demonstrate significantly more correct embedding of a face image in 3D than for all the baselines, further strengthened by a probably symmetric prior enabled during training.

OriginalspracheEnglisch
TitelProceedings - 2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024
Herausgeber (Verlag)Institute of Electrical and Electronics Engineers Inc.
Seiten3043-3053
Seitenumfang11
ISBN (elektronisch)9798350318920
DOIs
PublikationsstatusVeröffentlicht - 2024
Veranstaltung2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024 - Waikoloa, USA/Vereinigte Staaten
Dauer: 4 Jan. 20248 Jan. 2024

Publikationsreihe

NameProceedings - 2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024

Konferenz

Konferenz2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024
Land/GebietUSA/Vereinigte Staaten
OrtWaikoloa
Zeitraum4/01/248/01/24

Fingerprint

Untersuchen Sie die Forschungsthemen von „TriPlaneNet: An Encoder for EG3D Inversion“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren