# DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation

arXiv: https://arxiv.org/abs/2608.09288 · HTML: https://mlsanity.com/paper/2608.09288/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [Perfect match: Improved cross-modal embeddings for audio-visual synchronisation](https://mlsanity.com/paper/1809.08001/) (2018)
- [LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision](https://mlsanity.com/paper/2412.09262/) (2024)
- [CN-CELEB: a challenging Chinese speaker recognition dataset](https://mlsanity.com/paper/1911.01799/) (2019)
- [MossFormer2: Combining Transformer and RNN-Free Recurrent Network for Enhanced Time-Domain Monaural Speech Separation](https://mlsanity.com/paper/2312.11825/) (2023)
- [Time Domain Audio Visual Speech Separation](https://mlsanity.com/paper/1904.03760/) (2019)
- [TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation](https://mlsanity.com/paper/2410.01469/) (2024)
- [PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction](https://mlsanity.com/paper/2607.08111/) (2026)

## Abstract (arXiv)

Audio-visual speech enhancement under real-world conditions remains challenging due to unreliable visual inputs and the lack of large-scale training data with realistic acoustic conditions. Existing approaches usually fuse visual features directly into the separation network, making them vulnerable to degraded visual signals. In this paper, we present DAVE, a decoupled audio-visual enhancement framework for real-world speech separation. Firstly, to address the data scarcity issue, we construct DAVE-Corpus, a large-scale training corpus with 219,411 mixtures generated from public meeting corpora through combinatorial acoustic augmentation. Then, we introduce a progressive multi-objective optimization strategy to jointly improve speech separation, intelligibility, speaker identity preservation, and perceptual quality. We further develop a certified selective enhancement chain that applies scene routing, GAN-based denoising, and loudness normalization only within the no-reference partition, guaranteeing non-degradation of reference-based metrics. Experimental results on the Real-World Audio-Visual Speech Enhancement Challenge demonstrate the robustness of DAVE under both real-world mixed scenarios and visual degradation conditions.