# speech and noise dual-stream spectrogram refine network with speech distortion loss for robust speech recognition

arXiv: https://arxiv.org/abs/2305.17860 · HTML: https://mlsanity.com/paper/2305.17860/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 29 May 2023.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 16 of 16)

- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASR](https://mlsanity.com/paper/2201.06685/) (2022)
- [VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition](https://mlsanity.com/paper/2009.04323/) (2020)
- [Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Data](https://mlsanity.com/paper/2203.15321/) (2022)
- [Interactive Feature Fusion for End-to-End Noise-Robust Speech Recognition](https://mlsanity.com/paper/2110.05267/) (2021)
- [End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend](https://mlsanity.com/paper/2102.11525/) (2021)
- [Gated Recurrent Fusion with Joint Training Framework for Robust End-to-End Speech Recognition](https://mlsanity.com/paper/2011.04249/) (2020)
- [An Investigation of End-to-End Models for Robust Speech Recognition](https://mlsanity.com/paper/2102.06237/) (2021)
- [Multitask-Based Joint Learning Approach To Robust ASR For Radio Communication Speech](https://mlsanity.com/paper/2107.10701/) (2021)
- [End-to-end training of time domain audio separation and recognition](https://mlsanity.com/paper/1912.08462/) (2019)
- [Effect of noise suppression losses on speech distortion and ASR performance](https://mlsanity.com/paper/2111.11606/) (2021)
- [Improved Robustness to Disfluencies in RNN-Transducer Based Speech Recognition](https://mlsanity.com/paper/2012.06259/) (2020)
- [pMCT: Patched Multi-Condition Training for Robust Speech Recognition](https://mlsanity.com/paper/2207.04949/) (2022)
- [Mask scalar prediction for improving robust automatic speech recognition](https://mlsanity.com/paper/2204.12092/) (2022)

## Built on by (most cited first, 1 of 1)

- [ParaNoise-SV: Integrated Approach for Noise-Robust Speaker Verification with Parallel Joint Learning of Speech Enhancement and Noise Extraction](https://mlsanity.com/paper/2508.07219/) (2025)

## Abstract (arXiv)

In recent years, the joint training of speech enhancement front-end and automatic speech recognition (ASR) back-end has been widely used to improve the robustness of ASR systems. Traditional joint training methods only use enhanced speech as input for the backend. However, it is difficult for speech enhancement systems to directly separate speech from input due to the diverse types of noise with different intensities. Furthermore, speech distortion and residual noise are often observed in enhanced speech, and the distortion of speech and noise is different. Most existing methods focus on fusing enhanced and noisy features to address this issue. In this paper, we propose a dual-stream spectrogram refine network to simultaneously refine the speech and noise and decouple the noise from the noisy input. Our proposed method can achieve better performance with a relative 8.6% CER reduction.