mlsanityData to 7 Oct 2026

Audio-Driven Emotional Video Portraits

arXiv 2104.07452 ↗First posted 15 Apr 2021cs.CVcited by 180

Not on the map. Why?

Cited by 25 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite previous success in generating audio-driven talking heads, most of the previous studies focus on the correlation between speech content and the mouth shape. Facial emotion, which is one of the most important features on natural human faces, is always neglected in their methods. In this work, we present Emotional Video Portraits (EVP), a system for synthesizing high-quality video portraits with vivid emotional dynamics driven by audios. Specifically, we propose the Cross-Reconstructed Emotion Disentanglement technique to decompose speech into two decoupled spaces, i.e., a duration-independent emotion space and a duration dependent content space. With the disentangled features, dynamic 2D emotional facial landmarks can be deduced. Then we propose the Target-Adaptive Face Synthesis technique to generate the final high-quality video portraits, by bridging the gap between the deduced landmarks and the natural head poses of target videos. Extensive experiments demonstrate the effectiveness of our method both qualitatively and quantitatively.

Key facts. Cited by 180 papers in the map (data through 7 Oct 2026). Builds on 29 of its 50 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

29 of 50 references.

References · 29

  1. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  2. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  3. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  4. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  5. 2018Video-to-Video Synthesiscited by 388
  6. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  7. 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
  8. 2018Deep Video Portraitscited by 265
  9. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  10. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  11. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  12. 2018GANimation: Anatomically-aware Facial Animation from a Single Imagecited by 160
17 more
  1. 2018Lip Movements Generation at a Glancecited by 154
  2. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  3. 2018Look at Boundary: A Boundary-Aware Face Alignment Algorithmcited by 137
  4. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  5. 2019Text-based Editing of Talking-head Videocited by 102
  6. 2017You said that?cited by 99
  7. 2018Talking Face Generation by Conditional Recurrent Adversarial Networkcited by 84
  8. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  9. 2018VisemeNet: Audio-Driven Animator-Centric Speech Animationcited by 62
  10. 2019Frame attention networks for facial expression recognition in videoscited by 56
  11. 2017ExprGAN: Facial Expression Editing with Controllable Expression Intensitycited by 47
  12. 2019Neural Style-Preserving Visual Dubbingcited by 47
  13. 2019Learning Character-Agnostic Motion for Motion Retargeting in 2Dcited by 44
  14. 2020What comprises a good talking-head video generation?: A Survey and Benchmarkcited by 42
  15. 2020Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Imagescited by 37
  16. 2019Animating Face using Disentangled Audio Representationscited by 23
  17. 2019ET-GAN: Cross-Language Emotion Transfer Based on Cycle-Consistent Generative Adversarial Networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 29; 2023: 52; 2024: 40; 2025: 33; 2026: 17.

180 papers.

Sort:

By area:

  1. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  2. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  3. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  4. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  5. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  6. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  7. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 141
  8. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  9. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  10. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  11. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  12. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
38 more
  1. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  2. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 97
  3. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  4. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  5. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  6. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  7. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  8. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  9. 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
  10. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  11. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 73
  12. 2023OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Renderingcited by 67
  13. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  14. 2023Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head video Generationcited by 61
  15. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
  16. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priorscited by 55
  17. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  18. 2022SPACE: Speech-driven Portrait Animation with Controllable Expressioncited by 49
  19. 2023Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronizationcited by 49
  20. 2023RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatarscited by 45
  21. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  22. 2022Emotion-Controllable Generalized Talking Face Generationcited by 43
  23. 2023GAIA: Zero-shot Talking Avatar Generationcited by 43
  24. 2022Delving into Sequential Patches for Deepfake Detectioncited by 41
  25. 2023The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusioncited by 41
  26. 2023High-fidelity Generalized Emotional Talking Face Generation with Multi-modal Emotion Space Learningcited by 37
  27. 2021Live Speech Portraits: Real-Time Photorealistic Talking-Head Animationcited by 36
  28. 2022Sound-Guided Semantic Video Generationcited by 35
  29. 2022Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformerscited by 33
  30. 2024Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Stylecited by 33
  31. 2023TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Stylescited by 32
  32. 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
  33. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31
  34. 2021Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videoscited by 28
  35. 2022High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priorscited by 28
  36. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Modelcited by 28
  37. 2024Say Anything with Any Stylecited by 26
  38. 2025Human Motion Video Generation: A Surveycited by 25

50 most cited of 180 All 180 (JSON)