# Study of Lightweight Transformer Architectures for Single-Channel Speech Enhancement

arXiv: https://arxiv.org/abs/2505.21057 · HTML: https://mlsanity.com/paper/2505.21057/

## Key facts

- Cited by 3 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 27 May 2025.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 20 of 20)

- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Interspeech 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2101.01902/) (2021)
- [MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra](https://mlsanity.com/paper/2305.13686/) (2023)
- [DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering](https://mlsanity.com/paper/2110.05588/) (2021)
- [Towards efficient models for real-time deep noise suppression](https://mlsanity.com/paper/2101.09249/) (2021)
- [FRCRN: Boosting Feature Representation using Frequency Recurrence for Monaural Speech Enhancement](https://mlsanity.com/paper/2206.07293/) (2022)
- [CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement](https://mlsanity.com/paper/2209.11112/) (2022)
- [DeepFilterNet2: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio](https://mlsanity.com/paper/2205.05474/) (2022)
- [DPCRN: Dual-Path Convolution Recurrent Network for Single Channel Speech Enhancement](https://mlsanity.com/paper/2107.05429/) (2021)
- [On The Compensation Between Magnitude and Phase in Speech Separation](https://mlsanity.com/paper/2108.05470/) (2021)
- [DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement](https://mlsanity.com/paper/2305.08227/) (2023)
- [Perceptual Contrast Stretching on Target Feature for Speech Enhancement](https://mlsanity.com/paper/2203.17152/) (2022)
- [Multi-View Attention Transfer for Efficient Speech Enhancement](https://mlsanity.com/paper/2208.10367/) (2022)
- [Rethinking complex-valued deep neural networks for monaural speech enhancement](https://mlsanity.com/paper/2301.04320/) (2023)

## Built on by (most cited first, 3 of 3)

- [Dynamically Slimmable Speech Enhancement Network with Metric-Guided Training](https://mlsanity.com/paper/2510.11395/) (2025)
- [Advancing Electrolaryngeal Speech Enhancement Through Speech-Text Representation Learning](https://mlsanity.com/paper/2606.01905/) (2026)
- [Towards Robust Generative Speech Enhancement Using Vector Quantisation-Based Neural Audio Codec](https://mlsanity.com/paper/2606.16464/) (2026)

## Abstract (arXiv)

In speech enhancement, achieving state-of-the-art (SotA) performance while adhering to the computational constraints on edge devices remains a formidable challenge. Networks integrating stacked temporal and spectral modelling effectively leverage improved architectures such as transformers; however, they inevitably incur substantial computational complexity and model expansion. Through systematic ablation analysis on transformer-based temporal and spectral modelling, we demonstrate that the architecture employing streamlined Frequency-Time-Frequency (FTF) stacked transformers efficiently learns global dependencies within causal context, while avoiding considerable computational demands. Utilising discriminators in training further improves learning efficacy and enhancement without introducing additional complexity during inference. The proposed lightweight, causal, transformer-based architecture with adversarial training (LCT-GAN) yields SoTA performance on instrumental metrics among contemporary lightweight models, but with far less overhead. Compared to DeepFilterNet2, the LCT-GAN only requires 6% of the parameters, at similar complexity and performance. Against CCFNet+(Lite), LCT-GAN saves 9% in parameters and 10% in multiply-accumulate operations yet yielding improved performance. Further, the LCT-GAN even outperforms more complex, common baseline models on widely used test datasets.