# Deep Complex U-Net with Conformer for Audio-Visual Speech Enhancement

arXiv: https://arxiv.org/abs/2309.11059 · HTML: https://mlsanity.com/paper/2309.11059/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2023.

## Built on by (most cited first, 3 of 3)

- [Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement](https://mlsanity.com/paper/2501.13375/) (2025)
- [AUREXA-SE: Audio-Visual Unified Representation Exchange Architecture with Cross-Attention and Squeezeformer for Speech Enhancement](https://mlsanity.com/paper/2510.05295/) (2025)
- [Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2508.13624/) (2025)

## Abstract (arXiv)

Recent studies have increasingly acknowledged the advantages of incorporating visual data into speech enhancement (SE) systems. In this paper, we introduce a novel audio-visual SE approach, termed DCUC-Net (deep complex U-Net with conformer network). The proposed DCUC-Net leverages complex domain features and a stack of conformer blocks. The encoder and decoder of DCUC-Net are designed using a complex U-Net-based framework. The audio and visual signals are processed using a complex encoder and a ResNet-18 model, respectively. These processed signals are then fused using the conformer blocks and transformed into enhanced speech waveforms via a complex decoder. The conformer blocks consist of a combination of self-attention mechanisms and convolutional operations, enabling DCUC-Net to effectively capture both global and local audio-visual dependencies. Our experimental results demonstrate the effectiveness of DCUC-Net, as it outperforms the baseline model from the COG-MHEAR AVSE Challenge 2023 by a notable margin of 0.14 in terms of PESQ. Additionally, the proposed DCUC-Net performs comparably to a state-of-the-art model and outperforms all other compared models on the Taiwan Mandarin speech with video (TMSV) dataset.