mlsanityData to 7 Oct 2026

A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild

arXiv 2008.10010 ↗First posted 23 Aug 2020cs.CV cs.LG cs.SDcited by 669

Not on the map. Why?

Cited by 104 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we investigate the problem of lip-syncing a talking face video of an arbitrary identity to match a target speech segment. Current works excel at producing accurate lip movements on a static image or videos of specific people seen during the training phase. However, they fail to accurately morph the lip movements of arbitrary identities in dynamic, unconstrained talking face videos, resulting in significant parts of the video being out-of-sync with the new audio. We identify key reasons pertaining to this and hence resolve them by learning from a powerful lip-sync discriminator. Next, we propose new, rigorous evaluation benchmarks and metrics to accurately measure lip synchronization in unconstrained videos. Extensive quantitative evaluations on our challenging benchmarks show that the lip-sync accuracy of the videos generated by our Wav2Lip model is almost as good as real synced videos. We provide a demo video clearly showing the substantial impact of our Wav2Lip model and evaluation benchmarks on our website: urlcvit.iiit.ac.in/research/projects/cvit-projects/a-lip-sync-expert-is-all-you-need-for-speech-to-lip-generation-in-the-wild. The code and models are released at this GitHub repository: urlgithub.com/Rudrabha/Wav2Lip. You can also try out the interactive demo at this link: urlbhaasha.iiit.ac.in/lipsync.

Key facts. Cited by 669 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 68; 2023: 120; 2024: 150; 2025: 180; 2026: 118.

669 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  2. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  3. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  4. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  5. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  6. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  7. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  8. 2023Multimodal Large Language Models: A Surveycited by 193
  9. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  10. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  11. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  12. 2021FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Datasetcited by 179
38 more
  1. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  2. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  3. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  4. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  5. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  6. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  7. 2024DF40: Toward Next-Generation Deepfake Detectioncited by 126
  8. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
  9. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  10. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  11. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  12. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  13. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  14. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  15. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  16. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 97
  17. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  18. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  19. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  20. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  21. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  22. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  23. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  24. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  25. 2024V-Express: Conditional Dropout for Progressive Training of Portrait Video Generationcited by 85
  26. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  27. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  28. 2021Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forwardcited by 82
  29. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  30. 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
  31. 2021KoDF: A Large-scale Korean DeepFake Detection Datasetcited by 80
  32. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  33. 2021Deepfake Video Detection Using Convolutional Vision Transformercited by 79
  34. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
  35. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  36. 2022Object-Compositional Neural Implicit Surfacescited by 74
  37. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 73
  38. 2023Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesiscited by 72

50 most cited of 669 All 669 (JSON)