mlsanityData to 7 Oct 2026

MakeItTalk: Speaker-Aware Talking-Head Animation

arXiv 2004.12992 ↗First posted 27 Apr 2020cs.CV cs.GRcited by 310

Not on the map. Why?

Cited by 50 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a method that generates expressive talking heads from a single facial image with audio as the only input. In contrast to previous approaches that attempt to learn direct mappings from audio to raw pixels or points for creating talking faces, our method first disentangles the content and speaker information in the input audio signal. The audio content robustly controls the motion of lips and nearby facial regions, while the speaker information determines the specifics of facial expressions and the rest of the talking head dynamics. Another key component of our method is the prediction of facial landmarks reflecting speaker-aware dynamics. Based on this intermediate representation, our method is able to synthesize photorealistic videos of entire talking heads with full range of motion and also animate artistic paintings, sketches, 2D cartoon characters, Japanese mangas, stylized caricatures in a single unified framework. We present extensive quantitative and qualitative evaluation of our method, in addition to user studies, demonstrating generated talking heads of significantly higher quality compared to prior state-of-the-art.

Key facts. Cited by 310 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 44; 2023: 74; 2024: 81; 2025: 58; 2026: 31.

310 papers.

Sort:

By area:

  1. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  2. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  3. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  4. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  5. 2021Audio-Driven Emotional Video Portraitscited by 180
  6. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  7. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  8. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  9. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  10. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  11. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  12. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
38 more
  1. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  2. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  3. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  4. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  5. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  6. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  7. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 97
  8. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  9. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  10. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  11. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  12. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  13. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  14. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  15. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  16. 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
  17. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  18. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  19. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 73
  20. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 73
  21. 2023Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesiscited by 72
  22. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 67
  23. 2024AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encodingcited by 67
  24. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  25. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  26. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  27. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
  28. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priorscited by 55
  29. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  30. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  31. 2022SPACE: Speech-driven Portrait Animation with Controllable Expressioncited by 49
  32. 2023Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronizationcited by 49
  33. 2023DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencodercited by 48
  34. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  35. 2023VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Priorcited by 47
  36. 2021Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generationcited by 46
  37. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  38. 2022Emotion-Controllable Generalized Talking Face Generationcited by 43

50 most cited of 310 All 310 (JSON)