# On real-time multi-stage speech enhancement systems

arXiv: https://arxiv.org/abs/2312.12415 · HTML: https://mlsanity.com/paper/2312.12415/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 12 of 12)

- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement](https://mlsanity.com/paper/1709.08243/) (2017)
- [A Perceptually-Motivated Approach for Low-Complexity, Real-Time Enhancement of Fullband Speech](https://mlsanity.com/paper/2008.04259/) (2020)
- [DeepFilterNet2: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio](https://mlsanity.com/paper/2205.05474/) (2022)
- [A consolidated view of loss functions for supervised deep learning-based speech enhancement](https://mlsanity.com/paper/2009.12286/) (2020)
- [ICASSP 2021 Deep Noise Suppression Challenge: Decoupling Magnitude and Phase Optimization with a Two-Stage Deep Network](https://mlsanity.com/paper/2102.04198/) (2021)
- [DeepVQE: Real Time Deep Voice Quality Enhancement for Joint Acoustic Echo Cancellation, Noise Suppression and Dereverberation](https://mlsanity.com/paper/2306.03177/) (2023)
- [The NPU-Elevoc Personalized Speech Enhancement System for ICASSP2023 DNS Challenge](https://mlsanity.com/paper/2303.06811/) (2023)
- [Deep Multi-Frame Filtering for Hearing Aids](https://mlsanity.com/paper/2305.08225/) (2023)
- [EffCRN: An Efficient Convolutional Recurrent Network for High-Performance Speech Enhancement](https://mlsanity.com/paper/2306.02778/) (2023)

## Abstract (arXiv)

Recently, multi-stage systems have stood out among deep learning-based speech enhancement methods. However, these systems are always high in complexity, requiring millions of parameters and powerful computational resources, which limits their application for real-time processing in low-power devices. Besides, the contribution of various influencing factors to the success of multi-stage systems remains unclear, which presents challenges to reduce the size of these systems. In this paper, we extensively investigate a lightweight two-stage network with only 560k total parameters. It consists of a Mel-scale magnitude masking model in the first stage and a complex spectrum mapping model in the second stage. We first provide a consolidated view of the roles of gain power factor, post-filter, and training labels for the Mel-scale masking model. Then, we explore several training schemes for the two-stage network and provide some insights into the superiority of the two-stage network. We show that the proposed two-stage network trained by an optimal scheme achieves a performance similar to a four times larger open source model DeepFilterNet2.