mlsanityData to 14 Oct 2026

Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronization

arXiv 2308.09716 ↗First posted 18 Aug 2023cs.CV cs.AIcited by 49

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

The task of lip synchronization (lip-sync) seeks to match the lips of human faces with different audio. It has various applications in the film industry as well as for creating virtual avatars and for video conferencing. This is a challenging problem as one needs to simultaneously introduce detailed, realistic lip movements while preserving the identity, pose, emotions, and image quality. Many of the previous methods trying to solve this problem suffer from image quality degradation due to a lack of complete contextual information. In this paper, we present Diff2Lip, an audio-conditioned diffusion-based model which is able to do lip synchronization in-the-wild while preserving these qualities. We train our model on Voxceleb2, a video dataset containing in-the-wild talking face videos. Extensive studies show that our method outperforms popular methods like Wav2Lip and PC-AVS in Fr'echet inception distance (FID) metric and Mean Opinion Scores (MOS) of the users. We show results on both reconstruction (same audio-video inputs) as well as cross (different audio-video inputs) settings on Voxceleb2 and LRW datasets. Video results and code can be accessed from our project page ( https://soumik-kanad.github.io/diff2lip ).

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 44 of its 64 references that are in the map. First posted on arXiv 18 Aug 2023.

Builds on

44 of 64 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb
  2. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation10 Jan

Other references · 42

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2022Generative Adversarial Networkscited by 5,627
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
30 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  2. 2022Video Diffusion Modelscited by 2,305
  3. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  5. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  6. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  7. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 525
  8. 2018Video-to-Video Synthesiscited by 390
  9. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  10. 2018Everybody Dance Nowcited by 315
  11. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  12. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  13. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  14. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  15. 2021Audio-Driven Emotional Video Portraitscited by 180
  16. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  17. 2018Lip Movements Generation at a Glancecited by 155
  18. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  19. 2017You said that?cited by 99
  20. 2020Towards Automatic Face-to-Face Translationcited by 96
  21. 2018Talking Face Generation by Conditional Recurrent Adversarial Networkcited by 84
  22. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  23. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  24. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  25. 2021Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generationcited by 47
  26. 2021Live Speech Portraits: Real-Time Photorealistic Talking-Head Animationcited by 36
  27. 2022Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformerscited by 33
  28. 2018End-to-End Speech-Driven Facial Animation with Temporal GANscited by 31
  29. 2022Towards Realistic Visual Dubbing with Heterogeneous Sourcescited by 24
  30. 2021Diverse Video Generation using a Gaussian Process Triggercited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 28; 2026: 13.

49 papers.

Sort:

By area:

  1. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 235
  2. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 56
  3. 2025OmniSync: Towards Universal Lip Synchronization via Diffusion Transformerscited by 27
  4. 2023DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Modelscited by 16
  5. 2026Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillationcited by 13
  6. 2025SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusioncited by 12
  7. 2025KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolutioncited by 11
  8. 2025Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusioncited by 8
  9. 2025Detecting Lip-Syncing Deepfakes: Vision Temporal Transformer for Analyzing Mouth Inconsistenciescited by 7
  10. 2024The DeepSpeak Datasetcited by 6
  11. 2025Tell me Habibi, is it Real or Fake?cited by 6
  12. 2025PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysiscited by 5
37 more
  1. 2025FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesiscited by 3
  2. 2025Pindrop it! Audio and Visual Deepfake Countermeasures for Robust Detection and Fine Grained-Localizationcited by 3
  3. 2025DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detectioncited by 3
  4. 2025From Inpainting to Editing: Unlocking Robust Mask-Free Visual Dubbing via Generative Bootstrappingcited by 3
  5. 2024GLCF: A Global-Local Multimodal Coherence Analysis Framework for Talking Face Generation Detectioncited by 2
  6. 2025SynchroRaMa : Lip-Synchronized and Emotion-Aware Talking Face Generation via Multi-Modal Emotion Embeddingcited by 2
  7. 2025SyncAnyone: Implicit Disentanglement via Progressive Self-Correction for Lip-Syncing in the wildcited by 2
  8. 2026Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronizationcited by 2
  9. 2024High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Modelcited by 1
  10. 2025Removing Averaging: Personalized Lip-Sync Driven Characters Based on Identity Adaptercited by 1
  11. 2025RASA: Replace Anyone, Say Anything -- A Training-Free Framework for Audio-Driven and Universal Portrait Video Editingcited by 1
  12. 2025FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistencycited by 1
  13. 2025GenSync: A Generalized Talking Head Framework for Audio-driven Multi-Subject Lip-Sync using 3D Gaussian Splattingcited by 1
  14. 2025FauForensics: Boosting Audio-Visual Deepfake Detection with Facial Action Unitscited by 1
  15. 2025Video Editing for Audio-Visual Dubbingcited by 1
  16. 2025Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservationcited by 1
  17. 2025Performance Decay in Deepfake Detection: The Limitations of Training on Outdated Datacited by 1
  18. 2025Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Frameworkcited by 1
  19. 2025JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editingcited by 1
  20. 2026EfficientSync: Real-Time Lip Synchronization via Deformation-Based Reference Texture Mixingcited by 1
  21. 2024Superior and Pragmatic Talking Face Generation with Teacher-Student Frameworkcited by 0
  22. 2024SwapTalk: Audio-Driven Talking Face Generation with One-Shot Customization in Latent Spacecited by 0
  23. 2025StableDub: Taming Diffusion Prior for Generalized and Efficient Visual Dubbingcited by 0
  24. 2025R2M: Real-Aware Residual Model Merging for Robust and Generalizable Deepfake Detectioncited by 0
  25. 2025Shared Latent Representation for Joint Text-to-Audio-Visual Synthesiscited by 0
  26. 2025Asynchronous Pipeline Parallelism for Real-Time Multilingual Lip Synchronization in Video Communication Systemscited by 0
  27. 2025FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANscited by 0
  28. 2026UniSync: Towards Generalizable and High-Fidelity Lip Synchronization for Challenging Scenarioscited by 0
  29. 2026SyncEdit: Rethinking Lip Synchronization as Editing with Audio-Driven Diffusion Modelscited by 0
  30. 2026BioLip: Language-Generalizable Lip-Sync Deepfake Detection via Biomechanical Constraint Violation Modelingcited by 0
  31. 2026MoCoTalk: Multi-Conditional Diffusion with Adaptive Router for Controllable Talking Head Generationcited by 0
  32. 2026The Alpha Blending Hypothesis: Compositing Shortcut in Deepfake Detectioncited by 0
  33. 2026HighSync: High-Quality Lip Synchronization via Latent Diffusion Modelscited by 0
  34. 2026CUPID: Reconstructing UV Texture Maps for Interpretable Person-of-Interest Deepfake Detectioncited by 0
  35. 2026LoCC: Detection and Localization of Lip-Syncing Deepfakes via Counterfactual Frame Consistencycited by 0
  36. 2026ComplexSync: High-Fidelity and Real-Time Lip Sync in Complex Scenarioscited by 0
  37. 2026Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Style-Preserving Lip Sync via Audio-Aware Style Reference
  2. 2024Make Your Actor Talk: Generalizable and High-Fidelity Lip Sync with Motion and Appearance Disentanglement
  3. 2023SAiD: Speech-driven Blendshape Facial Animation with Diffusion
  4. 2024DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer
  5. 2025LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models
  6. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Model
  7. 2025Playmate2: Training-Free Multi-Character Audio-Driven Animation via Diffusion Transformer with Reward Feedback
  8. 2025Shushing! Let's Imagine an Authentic Speech from the Silent Video