mlsanityData to 14 Oct 2026

LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models

arXiv 2505.11391 ↗First posted 16 May 2025eess.AS cs.SDcited by 2

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We present LipDiffuser, a conditional diffusion model for lip-to-speech generation synthesizing natural and intelligible speech directly from silent video recordings. Our approach leverages the magnitude-preserving ablated diffusion model (MP-ADM) architecture as a denoiser model. To effectively condition the model, we incorporate visual features using magnitude-preserving feature-wise linear modulation (MP-FiLM) alongside speaker embeddings. A neural vocoder then reconstructs the speech waveform from the generated mel-spectrograms. Evaluations on LRS3 demonstrate that LipDiffuser outperforms existing lip-to-speech baselines in perceptual speech quality and speaker similarity, while remaining competitive in downstream automatic speech recognition. These findings are also supported by a formal listening experiment.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 49 of its 75 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

49 of 75 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Non-intrusive Speech Quality Assessment with Diffusion Models Trained on Clean Speech23 Oct
  2. 2024Diffusion Models for Audio Restoration15 Feb
  3. 2023DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding15 Aug
  4. 2023Audio-Visual Speech Enhancement with Score-Based Generative Models2 Jun
  5. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  12. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
31 more
  1. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  3. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  4. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  5. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  6. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  7. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  8. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  9. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  10. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
  11. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  12. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 162
  13. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  14. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  15. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 109
  16. 2020Lipreading using Temporal Convolutional Networkscited by 102
  17. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 97
  18. 2024A Survey on Diffusion Models for Time Series and Spatio-Temporal Datacited by 67
  19. 2024EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberationcited by 64
  20. 2024Temporally Aligned Audio for Video with Autoregressioncited by 52
  21. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  22. 2023Intelligible Lip-to-Speech Synthesis with Speech Unitscited by 28
  23. 2024Schr"odinger Bridge for Generative Speech Enhancementcited by 27
  24. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  25. 2023LipVoicer: Generating Speech from Silent Videos Guided by Lip Readingcited by 22
  26. 2023Lip-to-Speech Synthesis in the Wild with Multi-task Learningcited by 21
  27. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 21
  28. 2024Investigating Training Objectives for Generative Speech Enhancementcited by 20
  29. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 15
  30. 2022Insights Into Deep Non-linear Filters for Improved Multi-channel Speech Enhancementcited by 14
  31. 2025Do We Need EMA for Diffusion-Based Speech Enhancement? Toward a Magnitude-Preserving Network Architecturecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026C2FG: Control Classifier-Free Guidance via Score Discrepancy Analysiscited by 7
  2. 2026Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreadingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronization
  2. 2026SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesis
  3. 2025SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion
  4. 2024Style-Preserving Lip Sync via Audio-Aware Style Reference
  5. 2026TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation
  6. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Model
  7. 2025Shushing! Let's Imagine an Authentic Speech from the Silent Video
  8. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision