mlsanityData to 7 Oct 2026

ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding

arXiv 2207.09514 ↗First posted 19 Jul 2022eess.AS cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper presents recent progress on integrating speech separation and enhancement (SSE) into the ESPnet toolkit. Compared with the previous ESPnet-SE work, numerous features have been added, including recent state-of-the-art speech enhancement models with their respective training and evaluation recipes. Importantly, a new interface has been designed to flexibly combine speech enhancement front-ends with other tasks, including automatic speech recognition (ASR), speech translation (ST), and spoken language understanding (SLU). To showcase such integration, we performed experiments on carefully designed synthetic datasets for noisy-reverberant multi-channel ST and SLU tasks, which can be used as benchmark corpora for future research. In addition to these new tasks, we also use CHiME-4 and WSJ0-2Mix to benchmark multi- and single-channel SE approaches. Results show that the integration of SE front-ends with back-end tasks is a promising research direction even for tasks besides ASR, especially in the multi-channel scenario. The code is available online at https://github.com/ESPnet/ESPnet. The multi-channel ST and SLU datasets, which are another contribution of this work, are released on HuggingFace.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 40 of its 60 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

40 of 60 references.

References · 40

  1. 2018SDR - half-baked or well done?cited by 420
  2. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  3. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  4. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  5. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
  6. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  7. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  8. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  9. 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 133
  10. 2022Conditional Diffusion Probabilistic Model for Speech Enhancementcited by 132
  11. 2017Pyroomacoustics: A Python package for audio room simulations and array processing algorithmscited by 131
  12. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Resultscited by 128
28 more
  1. 2017TasNet: time-domain audio separation network for real-time, single-channel speech separationcited by 115
  2. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 97
  3. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  4. 2020Continuous speech separation: dataset and analysiscited by 73
  5. 2020Voice Separation with an Unknown Number of Multiple Speakerscited by 46
  6. 2020Asteroid: the PyTorch-based audio source separation toolkit for researcherscited by 45
  7. 2019MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognitioncited by 44
  8. 2021Interspeech 2021 Deep Noise Suppression Challengecited by 43
  9. 2018Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluationcited by 34
  10. 2021DPT-FSNet: Dual-path Transformer Based Full-band and Sub-band Fusion Network for Speech Enhancementcited by 33
  11. 2020ICASSP 2021 Deep Noise Suppression Challengecited by 27
  12. 2016Deep attractor network for single-microphone speaker separationcited by 26
  13. 2019FaSNet: Low-latency Adaptive Beamforming for Multi-microphone Audio Processingcited by 23
  14. 2020ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integrationcited by 22
  15. 2022SkiM: Skipping Memory LSTM for Low-Latency Real-Time Continuous Speech Separationcited by 16
  16. 2020Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASRcited by 15
  17. 2021End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontendcited by 12
  18. 2022L3DAS22 Challenge: Learning 3D Audio Sources in a Real Office Environmentcited by 10
  19. 2021Human Listening and Live Captioning: Multi-Task Training for Speech Enhancementcited by 7
  20. 2020Convolutive Transfer Function Invariant SDR training criteria for Multi-Channel Reverberant Speech Separationcited by 6
  21. 2021Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditionscited by 6
  22. 2020Surrogate Source Model Learning for Determined Source Separationcited by 5
  23. 2021SDR -- Medium Rare with Fast Computationscited by 4
  24. 2022Towards Low-distortion Multi-channel Speech Enhancement: The ESPNet-SE Submission to The L3DAS22 Challengecited by 4
  25. 2019Jointly optimal dereverberation and beamformingcited by 3
  26. 2021Leveraging Low-Distortion Target Estimates for Improved Speech Enhancementcited by 2
  27. 2017Rank-1 Constrained Multichannel Wiener Filter for Speech Recognition in Noisy Environmentscited by 1
  28. 2019Onssen: an open-source speech separation and enhancement librarycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 3; 2025: 1; 2026: 1.

8 papers.

  1. 2022Diffusion-based Generative Speech Source Separationcited by 41
  2. 2024Look Once to Hear: Target Speech Hearing with Noisy Examplescited by 11
  3. 2024Diffusion-based Generative Modeling with Discriminative Guidance for Streamable Speech Enhancementcited by 8
  4. 2023Exploring the Integration of Speech Separation and Recognition with Self-Supervised Learning Representationcited by 0
  5. 2023LC4SV: A Denoising Framework Learning to Compensate for Unseen Speaker Verification Modelscited by 0
  6. 2024ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integrationcited by 0
  7. 2025Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditionscited by 0
  8. 2026Multi-channel multi-speaker transformer for speech recognitioncited by 0