# Real-Time Audio-Visual End-to-End Speech Enhancement

arXiv: https://arxiv.org/abs/2303.07005 · HTML: https://mlsanity.com/paper/2303.07005/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2023.

## Built on by (most cited first, 5 of 5)

- [RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2407.07825/) (2024)
- [Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations](https://mlsanity.com/paper/2507.21448/) (2025)
- [SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer](https://mlsanity.com/paper/2505.04394/) (2025)
- [Real-Time System for Audio-Visual Target Speech Enhancement](https://mlsanity.com/paper/2509.20741/) (2025)
- [Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extraction](https://mlsanity.com/paper/2609.30631/) (2026)

## Abstract (arXiv)

Audio-visual speech enhancement (AV-SE) methods utilize auxiliary visual cues to enhance speakers' voices. Therefore, technically they should be able to outperform the audio-only speech enhancement (SE) methods. However, there are few works in the literature on an AV-SE system that can work in real time on a CPU. In this paper, we propose a low-latency real-time audio-visual end-to-end enhancement (AV-E3Net) model based on the recently proposed end-to-end enhancement network (E3Net). Our main contribution includes two aspects: 1) We employ a dense connection module to solve the performance degradation caused by the deep model structure. This module significantly improves the model's performance on the AV-SE task. 2) We propose a multi-stage gating-and-summation (GS) fusion module to merge audio and visual cues. Our results show that the proposed model provides better perceptual quality and intelligibility than the baseline E3net model with a negligible computational cost increase.