# TSTNN: Two-stage Transformer based Neural Network for Speech Enhancement in the Time Domain

arXiv: https://arxiv.org/abs/2103.09963 · HTML: https://mlsanity.com/paper/2103.09963/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 18 Mar 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement](https://mlsanity.com/paper/1905.04874/) (2019)
- [Phase-aware Speech Enhancement with Deep Complex U-Net](https://mlsanity.com/paper/1903.03107/) (2019)
- [Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation](https://mlsanity.com/paper/2007.13975/) (2020)
- [A Wavenet for Speech Denoising](https://mlsanity.com/paper/1706.07162/) (2017)
- [T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancement](https://mlsanity.com/paper/1910.06762/) (2019)
- [Self-Attentional Acoustic Models](https://mlsanity.com/paper/1803.09519/) (2018)
- [Improved Speech Enhancement with the Wave-U-Net](https://mlsanity.com/paper/1811.11307/) (2018)
- [Complex spectrogram enhancement by convolutional neural network with multi-metrics learning](https://mlsanity.com/paper/1704.08504/) (2017)

## Built on by (most cited first, 30 of 36)

- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [CMGAN: Conformer-based Metric GAN for Speech Enhancement](https://mlsanity.com/paper/2203.15149/) (2022)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement](https://mlsanity.com/paper/2209.11112/) (2022)
- [Dual-branch Attention-In-Attention Transformer for single-channel speech enhancement](https://mlsanity.com/paper/2110.06467/) (2021)
- [BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embedding](https://mlsanity.com/paper/2408.06851/) (2024)
- [Mamba-SEUNet: Mamba UNet for Monaural Speech Enhancement](https://mlsanity.com/paper/2412.16626/) (2024)
- [Multi-View Attention Transfer for Efficient Speech Enhancement](https://mlsanity.com/paper/2208.10367/) (2022)
- [BirdSoundsDenoising: Deep Visual Audio Denoising for Bird Sounds](https://mlsanity.com/paper/2210.10196/) (2022)
- [TrustEMG-Net: Using Representation-Masking Transformer with U-Net for Surface Electromyography Enhancement](https://mlsanity.com/paper/2410.03843/) (2024)
- [MUSE: Flexible Voiceprint Receptive Fields and Multi-Path Fusion Enhanced Taylor Transformer for U-Net-based Speech Enhancement](https://mlsanity.com/paper/2406.04589/) (2024)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [Complex Image Generation SwinTransformer Network for Audio Denoising](https://mlsanity.com/paper/2310.16109/) (2023)
- [DeFT-Mamba: Universal Multichannel Sound Separation and Polyphonic Audio Classification](https://mlsanity.com/paper/2409.12413/) (2024)
- [DENOASR: Debiasing ASRs through Selective Denoising](https://mlsanity.com/paper/2410.16712/) (2024)
- [Linguistic Knowledge Transfer Learning for Speech Enhancement](https://mlsanity.com/paper/2503.07078/) (2025)
- [Efficient Encoder-Decoder and Dual-Path Conformer for Comprehensive Feature Learning in Speech Enhancement](https://mlsanity.com/paper/2306.05861/) (2023)
- [DPATD: Dual-Phase Audio Transformer for Denoising](https://mlsanity.com/paper/2310.19588/) (2023)
- [Physics-Informed Data Denoising for Real-Life Sensing Systems](https://mlsanity.com/paper/2311.06968/) (2023)
- [ROSE: A Recognition-Oriented Speech Enhancement Framework in Air Traffic Control Using Multi-Objective Learning](https://mlsanity.com/paper/2312.06118/) (2023)
- [Audio Denoising for Robust Audio Fingerprinting](https://mlsanity.com/paper/2212.11277/) (2022)
- [Diffusion Gaussian Mixture Audio Denoise](https://mlsanity.com/paper/2406.09154/) (2024)
- [Throat and acoustic paired speech dataset for deep learning-based speech enhancement](https://mlsanity.com/paper/2502.11478/) (2025)
- [HiFi-Stream: Streaming Speech Enhancement with Generative Adversarial Networks](https://mlsanity.com/paper/2503.17141/) (2025)
- [A Low-Power Streaming Speech Enhancement Accelerator For Edge Devices](https://mlsanity.com/paper/2503.21335/) (2025)
- [Quaternion Self-Attention with Shared Scores](https://mlsanity.com/paper/2605.24920/) (2026)
- [Music Augmentation and Denoising For Peak-Based Audio Fingerprinting](https://mlsanity.com/paper/2310.13388/) (2023)
- [Improving Label Assignments Learning by Dynamic Sample Dropout Combined with Layer-wise Optimization in Speech Separation](https://mlsanity.com/paper/2311.12199/) (2023)
- [Survey: Transformer-based Models in Data Modality Conversion](https://mlsanity.com/paper/2408.04723/) (2024)

## Abstract (arXiv)

In this paper, we propose a transformer-based architecture, called two-stage transformer neural network (TSTNN) for end-to-end speech denoising in the time domain. The proposed model is composed of an encoder, a two-stage transformer module (TSTM), a masking module and a decoder. The encoder maps input noisy speech into feature representation. The TSTM exploits four stacked two-stage transformer blocks to efficiently extract local and global information from the encoder output stage by stage. The masking module creates a mask which will be multiplied with the encoder output. Finally, the decoder uses the masked encoder feature to reconstruct the enhanced speech. Experimental results on the benchmark dataset show that the TSTNN outperforms most state-of-the-art models in time or frequency domain while having significantly lower model complexity.