# Dual-branch Attention-In-Attention Transformer for single-channel speech enhancement

arXiv: https://arxiv.org/abs/2110.06467 · HTML: https://mlsanity.com/paper/2110.06467/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 13 Oct 2021.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement](https://mlsanity.com/paper/2104.03538/) (2021)
- [MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement](https://mlsanity.com/paper/1905.04874/) (2019)
- [Phase-aware Speech Enhancement with Deep Complex U-Net](https://mlsanity.com/paper/1903.03107/) (2019)
- [Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation](https://mlsanity.com/paper/2007.13975/) (2020)
- [PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network](https://mlsanity.com/paper/1911.04697/) (2019)
- [TSTNN: Two-stage Transformer based Neural Network for Speech Enhancement in the Time Domain](https://mlsanity.com/paper/2103.09963/) (2021)
- [T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancement](https://mlsanity.com/paper/1910.06762/) (2019)
- [Speech Enhancement using Self-Adaptation and Multi-Head Self-Attention](https://mlsanity.com/paper/2002.05873/) (2020)
- [Glance and Gaze: A Collaborative Learning Framework for Single-channel Speech Enhancement](https://mlsanity.com/paper/2106.11789/) (2021)
- [On The Compensation Between Magnitude and Phase in Speech Separation](https://mlsanity.com/paper/2108.05470/) (2021)
- [A Simultaneous Denoising and Dereverberation Framework with Target Decoupling](https://mlsanity.com/paper/2106.12743/) (2021)
- [CycleGAN-based Non-parallel Speech Enhancement with an Adaptive Attention-in-attention Mechanism](https://mlsanity.com/paper/2107.13143/) (2021)
- [On Loss Functions and Recurrency Training for GAN-based Speech Enhancement Systems](https://mlsanity.com/paper/2007.14974/) (2020)
- [Deep Residual-Dense Lattice Network for Speech Enhancement](https://mlsanity.com/paper/2002.12794/) (2020)

## Built on by (most cited first, 18 of 18)

- [HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement](https://mlsanity.com/paper/2203.13086/) (2022)
- [LEMON: Learning 3D Human-Object Interaction Relation from 2D Images](https://mlsanity.com/paper/2312.08963/) (2023)
- [CMGAN: Conformer-based Metric GAN for Speech Enhancement](https://mlsanity.com/paper/2203.15149/) (2022)
- [CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement](https://mlsanity.com/paper/2209.11112/) (2022)
- [Mamba-SEUNet: Mamba UNet for Monaural Speech Enhancement](https://mlsanity.com/paper/2412.16626/) (2024)
- [SCP-GAN: Self-Correcting Discriminator Optimization for Training Consistency Preserving Metric GAN on Speech Enhancement Tasks](https://mlsanity.com/paper/2210.14474/) (2022)
- [FINALLY: fast and universal speech enhancement with studio-like quality](https://mlsanity.com/paper/2410.05920/) (2024)
- [Speech Enhancement Using Continuous Embeddings of Neural Audio Codec](https://mlsanity.com/paper/2502.16240/) (2025)
- [Exploring Self-Attention Mechanisms for Speech Separation](https://mlsanity.com/paper/2202.02884/) (2022)
- [MUSE: Flexible Voiceprint Receptive Fields and Multi-Path Fusion Enhanced Taylor Transformer for U-Net-based Speech Enhancement](https://mlsanity.com/paper/2406.04589/) (2024)
- [A Survey of Deep Learning for Complex Speech Spectrograms](https://mlsanity.com/paper/2505.08694/) (2025)
- [Efficient Encoder-Decoder and Dual-Path Conformer for Comprehensive Feature Learning in Speech Enhancement](https://mlsanity.com/paper/2306.05861/) (2023)
- [ROSE: A Recognition-Oriented Speech Enhancement Framework in Air Traffic Control Using Multi-Objective Learning](https://mlsanity.com/paper/2312.06118/) (2023)
- [Audio Denoising for Robust Audio Fingerprinting](https://mlsanity.com/paper/2212.11277/) (2022)
- [Diffusion Gaussian Mixture Audio Denoise](https://mlsanity.com/paper/2406.09154/) (2024)
- [Music Augmentation and Denoising For Peak-Based Audio Fingerprinting](https://mlsanity.com/paper/2310.13388/) (2023)
- [IMSE: Efficient U-Net-based Speech Enhancement using Inception Depthwise Convolution and Amplitude-Aware Linear Attention](https://mlsanity.com/paper/2511.14515/) (2025)
- [Neuromorphic Speech Enhancement with Dual-Branch Spiking Neural Networks](https://mlsanity.com/paper/2606.23761/) (2026)

## Abstract (arXiv)

Curriculum learning begins to thrive in the speech enhancement area, which decouples the original spectrum estimation task into multiple easier sub-tasks to achieve better performance. Motivated by that, we propose a dual-branch attention-in-attention transformer dubbed DB-AIAT to handle both coarse- and fine-grained regions of the spectrum in parallel. From a complementary perspective, a magnitude masking branch is proposed to coarsely estimate the overall magnitude spectrum, and simultaneously a complex refining branch is elaborately designed to compensate for the missing spectral details and implicitly derive phase information. Within each branch, we propose a novel attention-in-attention transformer-based module to replace the conventional RNNs and temporal convolutional networks for temporal sequence modeling. Specifically, the proposed attention-in-attention transformer consists of adaptive temporal-frequency attention transformer blocks and an adaptive hierarchical attention module, aiming to capture long-term temporal-frequency dependencies and further aggregate global hierarchical contextual information. Experimental results on Voice Bank + DEMAND demonstrate that DB-AIAT yields state-of-the-art performance (e.g., 3.31 PESQ, 95.6% STOI and 10.79dB SSNR) over previous advanced systems with a relatively small model size (2.81M).