# Diffusion-based Unsupervised Audio-visual Speech Enhancement

arXiv: https://arxiv.org/abs/2410.05301 · HTML: https://mlsanity.com/paper/2410.05301/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 30 of 30)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation](https://mlsanity.com/paper/2303.00848/) (2023)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020)
- [Universal Speech Enhancement with Score-based Diffusion](https://mlsanity.com/paper/2206.03065/) (2022)
- [Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks](https://mlsanity.com/paper/1709.00944/) (2017)
- [Cold Diffusion for Speech Enhancement](https://mlsanity.com/paper/2211.02527/) (2022)
- [Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders](https://mlsanity.com/paper/2106.12271/) (2021)
- [Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoders](https://mlsanity.com/paper/1908.02590/) (2019)
- [My lips are concealed: Audio-visual speech enhancement through obstructions](https://mlsanity.com/paper/1907.04975/) (2019)
- [FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching](https://mlsanity.com/paper/2406.09286/) (2024)
- [LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading](https://mlsanity.com/paper/2306.03258/) (2023)
- [Improved Lite Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2008.13222/) (2020)
- [Audio-Visual Speech Enhancement with Score-Based Generative Models](https://mlsanity.com/paper/2306.01432/) (2023)
- [Unsupervised speech enhancement with diffusion-based generative models](https://mlsanity.com/paper/2309.10450/) (2023)
- [Diffusion-Based Speech Enhancement in Matched and Mismatched Conditions Using a Heun-Based Sampler](https://mlsanity.com/paper/2312.02683/) (2023)
- [Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2105.08970/) (2021)
- [Audio-Visual Speech Enhancement and Separation by Utilizing Multi-Modal Self-Supervised Embeddings](https://mlsanity.com/paper/2210.17456/) (2022)
- [AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2309.08030/) (2023)
- [The Effect of Training Dataset Size on Discriminative and Diffusion-Based Speech Enhancement Systems](https://mlsanity.com/paper/2406.06160/) (2024)
- [Audio-visual speech enhancement with a deep Kalman filter generative model](https://mlsanity.com/paper/2211.00988/) (2022)
- [Unsupervised speech enhancement with deep dynamical generative speech and noise models](https://mlsanity.com/paper/2306.07820/) (2023)

## Built on by (most cited first, 8 of 8)

- [Diffusion-Based Unsupervised Audio-Visual Speech Separation in Noisy Environments with Noise Prior](https://mlsanity.com/paper/2509.14379/) (2025)
- [Conditional Flow Matching for Visually-Guided Acoustic Highlighting](https://mlsanity.com/paper/2602.03762/) (2026)
- [Arch-VQ: Discrete Architecture Representation Learning with Autoregressive Priors](https://mlsanity.com/paper/2503.22063/) (2025)
- [Posterior Transition Modeling for Unsupervised Diffusion-Based Speech Enhancement](https://mlsanity.com/paper/2507.02391/) (2025)
- [SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling](https://mlsanity.com/paper/2602.01394/) (2026)
- [Audio-visual Contrastive Alignment for Diffusion-based Visual-conditioned Speech Enhancement](https://mlsanity.com/paper/2606.23712/) (2026)
- [SG-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2609.18009/) (2026)
- [Exposing the Cost of Deep Learning Audio Development](https://mlsanity.com/paper/2610.01619/) (2026)

## Abstract (arXiv)

This paper proposes a new unsupervised audio-visual speech enhancement (AVSE) approach that combines a diffusion-based audio-visual speech generative model with a non-negative matrix factorization (NMF) noise model. First, the diffusion model is pre-trained on clean speech conditioned on corresponding video data to simulate the speech generative distribution. This pre-trained model is then paired with the NMF-based noise model to estimate clean speech iteratively. Specifically, a diffusion-based posterior sampling approach is implemented within the reverse diffusion process, where after each iteration, a speech estimate is obtained and used to update the noise parameters. Experimental results confirm that the proposed AVSE approach not only outperforms its audio-only counterpart but also generalizes better than a recent supervised-generative AVSE method. Additionally, the new inference algorithm offers a better balance between inference speed and performance compared to the previous diffusion-based method. Code and demo available at: https://jeaneudesayilo.github.io/fast_UdiffSE