# Diffusion-Based Unsupervised Audio-Visual Speech Separation in Noisy Environments with Noise Prior

arXiv: https://arxiv.org/abs/2509.14379 · HTML: https://mlsanity.com/paper/2509.14379/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 19 of 19)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Solving Audio Inverse Problems with a Diffusion Model](https://mlsanity.com/paper/2210.15228/) (2022)
- [FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching](https://mlsanity.com/paper/2406.09286/) (2024)
- [Blind Audio Bandwidth Extension: A Diffusion-Based Zero-Shot Approach](https://mlsanity.com/paper/2306.01433/) (2023)
- [BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition](https://mlsanity.com/paper/2404.02098/) (2024)
- [Unsupervised speech enhancement with diffusion-based generative models](https://mlsanity.com/paper/2309.10450/) (2023)
- [BUDDy: Single-Channel Blind Unsupervised Dereverberation with Diffusion Models](https://mlsanity.com/paper/2405.04272/) (2024)
- [Diffusion-based Unsupervised Audio-visual Speech Enhancement](https://mlsanity.com/paper/2410.05301/) (2024)
- [ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior](https://mlsanity.com/paper/2505.05657/) (2025)

## Built on by (most cited first, 1 of 1)

- [SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling](https://mlsanity.com/paper/2602.01394/) (2026)

## Abstract (arXiv)

In this paper, we address the problem of single-microphone speech separation in the presence of ambient noise. We propose a generative unsupervised technique that directly models both clean speech and structured noise components, training exclusively on these individual signals rather than noisy mixtures. Our approach leverages an audio-visual score model that incorporates visual cues to serve as a strong generative speech prior. By explicitly modelling the noise distribution alongside the speech distribution, we enable effective decomposition through the inverse problem paradigm. We perform speech separation by sampling from the posterior distributions via a reverse diffusion process, which directly estimates and removes the modelled noise component to recover clean constituent signals. Experimental results demonstrate promising performance, highlighting the effectiveness of our direct noise modelling approach in challenging acoustic environments.