# Audio-Visual Speech Enhancement In Complex Scenarios With Separation And Dereverberation Joint Modeling

arXiv: https://arxiv.org/abs/2510.26825 · HTML: https://mlsanity.com/paper/2510.26825/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 20 of 20)

- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020)
- [TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation](https://mlsanity.com/paper/2211.12433/) (2022)
- [SpEx+: A Complete Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2005.04686/) (2020)
- [Time Domain Audio Visual Speech Separation](https://mlsanity.com/paper/1904.03760/) (2019)
- [SpEx: Multi-Scale Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2004.08326/) (2020)
- [Muse: Multi-modal target speaker extraction with visual cues](https://mlsanity.com/paper/2010.07775/) (2020)
- [Multi-modal Multi-channel Target Speech Separation](https://mlsanity.com/paper/2003.07032/) (2020)
- [SkipConvNet: Skip Convolutional Neural Network for Speech Dereverberation using Optimally Smoothed Spectral Mapping](https://mlsanity.com/paper/2007.09131/) (2020)
- [USEV: Universal Speaker Extraction with Visual Cue](https://mlsanity.com/paper/2109.14831/) (2021)
- [IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation](https://mlsanity.com/paper/2308.08143/) (2023)
- [An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits](https://mlsanity.com/paper/2212.10744/) (2022)
- [A Hybrid Continuity Loss to Reduce Over-Suppression for Time-domain Target Speaker Extraction](https://mlsanity.com/paper/2203.16843/) (2022)
- [Target Confusion in End-to-end Speaker Extraction: Analysis and Approaches](https://mlsanity.com/paper/2204.01355/) (2022)
- [Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction](https://mlsanity.com/paper/2310.19644/) (2023)

## Abstract (arXiv)

Audio-visual speech enhancement (AVSE) is a task that uses visual auxiliary information to extract a target speaker's speech from mixed audio. In real-world scenarios, there often exist complex acoustic environments, accompanied by various interfering sounds and reverberation. Most previous methods struggle to cope with such complex conditions, resulting in poor perceptual quality of the extracted speech. In this paper, we propose an effective AVSE system that performs well in complex acoustic environments. Specifically, we design a "separation before dereverberation" pipeline that can be extended to other AVSE networks. The 4th COGMHEAR Audio-Visual Speech Enhancement Challenge (AVSEC) aims to explore new approaches to speech processing in multimodal complex environments. We validated the performance of our system in AVSEC-4: we achieved excellent results in the three objective metrics on the competition leaderboard, and ultimately secured first place in the human subjective listening test.