TY - JOUR
T1 - Real-time expression transfer for facial reenactment
AU - Thies, Justus
AU - Zollhöfer, Michael
AU - Nießner, Matthias
AU - Valgaerts, Levi
AU - Stamminger, Marc
AU - Theobalt, Christian
N1 - Publisher Copyright:
Copyright 2015 ACM.
PY - 2015/11
Y1 - 2015/11
N2 - We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor. The novelty of our approach lies in the transfer and photorealistic re-rendering of facial deformations and detail into the target video in a way that the newly-synthesized expressions are virtually indistinguishable from a real video. To achieve this, we accurately capture the facial performances of the source and target subjects in real-time using a commodity RGB-D sensor. For each frame, we jointly fit a parametric model for identity, expression, and skin reflectance to the input color and depth data, and also reconstruct the scene lighting. For expression transfer, we compute the difference between the source and target expressions in parameter space, and modify the target parameters to match the source expressions. A major challenge is the convincing re-rendering of the synthesized target face into the corresponding video stream. This requires a careful consideration of the lighting and shading design, which both must correspond to the real-world environment. We demonstrate our method in a live setup, where we modify a video conference feed such that the facial expressions of a different person (e.g., translator) are matched in real-time.
AB - We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor. The novelty of our approach lies in the transfer and photorealistic re-rendering of facial deformations and detail into the target video in a way that the newly-synthesized expressions are virtually indistinguishable from a real video. To achieve this, we accurately capture the facial performances of the source and target subjects in real-time using a commodity RGB-D sensor. For each frame, we jointly fit a parametric model for identity, expression, and skin reflectance to the input color and depth data, and also reconstruct the scene lighting. For expression transfer, we compute the difference between the source and target expressions in parameter space, and modify the target parameters to match the source expressions. A major challenge is the convincing re-rendering of the synthesized target face into the corresponding video stream. This requires a careful consideration of the lighting and shading design, which both must correspond to the real-world environment. We demonstrate our method in a live setup, where we modify a video conference feed such that the facial expressions of a different person (e.g., translator) are matched in real-time.
KW - Depth camera
KW - Expression transfer
KW - Faces
KW - Real-time
UR - http://www.scopus.com/inward/record.url?scp=84995921764&partnerID=8YFLogxK
U2 - 10.1145/2816795.2818056
DO - 10.1145/2816795.2818056
M3 - Article
AN - SCOPUS:84995921764
SN - 0730-0301
VL - 34
JO - ACM Transactions on Graphics
JF - ACM Transactions on Graphics
IS - 6
M1 - 183
ER -