# U-Former: Improving Monaural Speech Enhancement with Multi-head Self and Cross Attention

arXiv: https://arxiv.org/abs/2205.08681 · HTML: https://mlsanity.com/paper/2205.08681/

## Key facts

- Cited by 3 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 18 May 2022.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://mlsanity.com/paper/1803.04831/) (2018)
- [Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation](https://mlsanity.com/paper/2007.13975/) (2020)
- [A Wavenet for Speech Denoising](https://mlsanity.com/paper/1706.07162/) (2017)
- [TSTNN: Two-stage Transformer based Neural Network for Speech Enhancement in the Time Domain](https://mlsanity.com/paper/2103.09963/) (2021)
- [T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancement](https://mlsanity.com/paper/1910.06762/) (2019)
- [A Fully Convolutional Neural Network for Speech Enhancement](https://mlsanity.com/paper/1609.07132/) (2016)
- [End-to-End Multi-Channel Speech Separation](https://mlsanity.com/paper/1905.06286/) (2019)
- [Supervised and Unsupervised Speech Enhancement Using Nonnegative Matrix Factorization](https://mlsanity.com/paper/1709.05362/) (2017)
- [Monaural Speech Enhancement with Complex Convolutional Block Attention Module and Joint Time Frequency Losses](https://mlsanity.com/paper/2102.01993/) (2021)

## Built on by (most cited first, 3 of 3)

- [PCNN: A Lightweight Parallel Conformer Neural Network for Efficient Monaural Speech Enhancement](https://mlsanity.com/paper/2307.15251/) (2023)
- [All Information is Necessary: Integrating Speech Positive and Negative Information by Contrastive Learning for Speech Enhancement](https://mlsanity.com/paper/2304.13439/) (2023)
- [SE Territory: Monaural Speech Enhancement Meets the Fixed Virtual Perceptual Space Mapping](https://mlsanity.com/paper/2311.01679/) (2023)

## Abstract (arXiv)

For supervised speech enhancement, contextual information is important for accurate spectral mapping. However, commonly used deep neural networks (DNNs) are limited in capturing temporal contexts. To leverage long-term contexts for tracking a target speaker, this paper treats the speech enhancement as sequence-to-sequence mapping, and propose a novel monaural speech enhancement U-net structure based on Transformer, dubbed U-Former. The key idea is to model long-term correlations and dependencies, which are crucial for accurate noisy speech modeling, through the multi-head attention mechanisms. For this purpose, U-Former incorporates multi-head attention mechanisms at two levels: 1) a multi-head self-attention module which calculate the attention map along both time- and frequency-axis to generate time and frequency sub-attention maps for leveraging global interactions between encoder features, while 2) multi-head cross-attention module which are inserted in the skip connections allows a fine recovery in the decoder by filtering out uncorrelated features. Experimental results illustrate that the U-Former obtains consistently better performance than recent models of PESQ, STOI, and SSNR scores.