Xin Wen1    Miao Wang1,2    Christian Richardt3    Ze-Yin Chen1    Shi-Min Hu4

1 Beihang University       2 Peng Cheng Laboratory       3 University of Bath       4 Tsinghua University

IEEE Transactions on Visualization and Computer Graphics (ISMAR 2020)


Abstract

Video portraits are common in a variety of applications, such as videoconferencing, news broadcasting, and virtual education and training. We present a novel method to synthesize photorealistic video portraits for an input portrait video, automatically driven by a person’s voice. The main challenge in this task is the hallucination of plausible, photorealistic facial expressions from input speech audio. To address this challenge, we employ a parametric 3D face model represented by geometry, facial expression, illumination, etc., and learn a mapping from audio features to model parameters. The input source audio is first represented as a high-dimensional feature, which is used to predict facial expression parameters of the 3D face model. We then replace the expression parameters computed from the original target video with the predicted one, and rerender the reenacted face. Finally, we generate a photorealistic video portrait from the reenacted synthetic face sequence via a neural face renderer. One appealing feature of our approach is the generalization capability for various input speech audio, including synthetic speech audio from text-to-speech software. Extensive experimental results show that our approach outperforms previous general-purpose audio-driven video portrait methods. This includes a user study demonstrating that our results are rated as more realistic than previous methods.

Downloads

Bibtex

@article{AudioDVP,
  author  = {Xin Wen and Miao Wang and Christian Richardt and Ze-Yin Chen and Shi-Min Hu},
  title   = {Photorealistic Audio-driven Video Portraits},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year    = {2020},
  volume  = {26},
  number  = {12},
  month   = dec,
  pages   = {3457--3466},
  doi     = {10.1109/TVCG.2020.3023573},
  url     = {https://richardt.name/audio-dvp/},
}