mlsanityData to 14 Oct 2026

SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation

arXiv 2211.12194 ↗First posted 22 Nov 2022cs.CVcited by 382

Not on the map. Why?

Cited by 84 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating talking head videos through a face image and a piece of speech audio still contains many challenges. ie, unnatural head movement, distorted expression, and identity modification. We argue that these issues are mainly because of learning from the coupled 2D motion fields. On the other hand, explicitly using 3D information also suffers problems of stiff expression and incoherent video. We present SadTalker, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation. To learn the realistic motion coefficients, we explicitly model the connections between audio and different types of motion coefficients individually. Precisely, we present ExpNet to learn the accurate facial expression from audio by distilling both coefficients and 3D-rendered faces. As for the head pose, we design PoseVAE via a conditional VAE to synthesize head motion in different styles. Finally, the generated 3D motion coefficients are mapped to the unsupervised 3D keypoints space of the proposed face render, and synthesize the final video. We conducted extensive experiments to demonstrate the superiority of our method in terms of motion and video quality.

Key facts. Cited by 382 papers in the map (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

41 of 55 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  7. 2020First Order Motion Model for Image Animationcited by 704
  8. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  9. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 482
  10. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 444
  11. 2016Tutorial on Variational Autoencoderscited by 409
  12. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 349
29 more
  1. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  2. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  3. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  4. 2018Deep Video Portraitscited by 267
  5. 2021Motion Representations for Articulated Animationcited by 255
  6. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  7. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  8. 2021HeadNeRF: A Real-time NeRF-based Parametric Head Modelcited by 228
  9. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 212
  10. 2022Thin-Plate Spline Motion Model for Image Animationcited by 210
  11. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
  12. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 203
  13. 2019Few-shot Video-to-Video Synthesiscited by 197
  14. 2019Learning Individual Styles of Conversational Gesturecited by 191
  15. 2021Audio-Driven Emotional Video Portraitscited by 180
  16. 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 178
  17. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 175
  18. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  19. 2017Fine-Grained Head Pose Estimation Without Keypointscited by 163
  20. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  21. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 149
  22. 2020HeadGAN: One-shot Neural Head Synthesis and Editingcited by 105
  23. 2020Towards Automatic Face-to-Face Translationcited by 96
  24. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  25. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  26. 2022Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videoscited by 50
  27. 2022Implicit Warping for Animation with Image Setscited by 45
  28. 2021Live Speech Portraits: Real-Time Photorealistic Talking-Head Animationcited by 36
  29. 2022Training Strategies for Improved Lip-readingcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 39; 2024: 106; 2025: 149; 2026: 87.

382 papers.

Sort:

By area:

  1. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  2. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 328
  3. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 235
  4. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
  5. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  6. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 194
  7. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 187
  8. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 174
  9. 2024DF40: Toward Next-Generation Deepfake Detectioncited by 129
  10. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 129
  11. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 122
  12. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
38 more
  1. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 97
  2. 2024Generalizable and Animatable Gaussian Head Avatarcited by 97
  3. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 89
  4. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  5. 2025OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animationcited by 83
  6. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  7. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 80
  8. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  9. 2023PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalizationcited by 77
  10. 2024GPAvatar: Generalizable and Precise Head Avatar from Image(s)cited by 74
  11. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 72
  12. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 72
  13. 2025HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characterscited by 69
  14. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 67
  15. 2024EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatarscited by 66
  16. 2025Let Them Talk: Audio-Driven Multi-Person Conversational Video Generationcited by 66
  17. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 62
  18. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 50
  19. 2023VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Priorcited by 47
  20. 2025StableAvatar: Infinite-Length Audio-Driven Avatar Video Generationcited by 44
  21. 2023GAIA: Zero-shot Talking Avatar Generationcited by 43
  22. 2025LAM: Large Avatar Model for One-shot Animatable Gaussian Headcited by 43
  23. 2025Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animationcited by 43
  24. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 41
  25. 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesiscited by 41
  26. 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generationcited by 40
  27. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  28. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
  29. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  30. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantizationcited by 39
  31. 2024CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attentioncited by 39
  32. 2024MEMO: Memory-Guided Diffusion for Expressive Talking Video Generationcited by 38
  33. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  34. 2024Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Stylecited by 33
  35. 2023TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Stylescited by 32
  36. 2024Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakescited by 32
  37. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 32
  38. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31

50 most cited of 382 All 382 (JSON)