# Towards Robust Real-time Audio-Visual Speech Enhancement

arXiv: https://arxiv.org/abs/2112.09060 · HTML: https://mlsanity.com/paper/2112.09060/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 17 of 17)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Resnet in Resnet: Generalizing Residual Architectures](https://mlsanity.com/paper/1603.08029/) (2016)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks](https://mlsanity.com/paper/1709.00944/) (2017)
- [My lips are concealed: Audio-visual speech enhancement through obstructions](https://mlsanity.com/paper/1907.04975/) (2019)
- [Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition](https://mlsanity.com/paper/1711.05747/) (2017)
- [Lip-Reading Driven Deep Learning Approach for Speech Enhancement](https://mlsanity.com/paper/1808.00046/) (2018)
- [A cappella: Audio-visual Singing Voice Separation](https://mlsanity.com/paper/2104.09946/) (2021)
- [DNN driven Speaker Independent Audio-Visual Mask Estimation for Speech Separation](https://mlsanity.com/paper/1808.00060/) (2018)
- [Deep-Learning-Based Audio-Visual Speech Enhancement in Presence of Lombard Effect](https://mlsanity.com/paper/1905.12605/) (2019)
- [On the Role of Visual Cues in Audiovisual Speech Enhancement](https://mlsanity.com/paper/2004.12031/) (2020)

## Built on by (most cited first, 2 of 2)

- [RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2407.07825/) (2024)
- [On Robustness to Missing Video for Audiovisual Speech Recognition](https://mlsanity.com/paper/2312.10088/) (2023)

## Abstract (arXiv)

The human brain contextually exploits heterogeneous sensory information to efficiently perform cognitive tasks including vision and hearing. For example, during the cocktail party situation, the human auditory cortex contextually integrates audio-visual (AV) cues in order to better perceive speech. Recent studies have shown that AV speech enhancement (SE) models can significantly improve speech quality and intelligibility in very low signal to noise ratio (SNR) environments as compared to audio-only SE models. However, despite significant research in the area of AV SE, development of real-time processing models with low latency remains a formidable technical challenge. In this paper, we present a novel framework for low latency speaker-independent AV SE that can generalise on a range of visual and acoustic noises. In particular, a generative adversarial networks (GAN) is proposed to address the practical issue of visual imperfections in AV SE. In addition, we propose a deep neural network based real-time AV SE model that takes into account the cleaned visual speech output from GAN to deliver more robust SE. The proposed framework is evaluated on synthetic and real noisy AV corpora using objective speech quality and intelligibility metrics and subjective listing tests. Comparative simulation results show that our real time AV SE framework outperforms state-of-the-art SE approaches, including recent DNN based SE models.