# ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integration

arXiv: https://arxiv.org/abs/2011.03706 · HTML: https://mlsanity.com/paper/2011.03706/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2020.

## Built on by (most cited first, 22 of 22)

- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [WaveFake: A Data Set to Facilitate Audio Deepfake Detection](https://mlsanity.com/paper/2111.02813/) (2021)
- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.04590/) (2021)
- [AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement](https://mlsanity.com/paper/2501.15417/) (2025)
- [Rescoring Sequence-to-Sequence Models for Text Line Recognition with CTC-Prefixes](https://mlsanity.com/paper/2110.05909/) (2021)
- [Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion](https://mlsanity.com/paper/2208.08757/) (2022)
- [Annealed Multiple Choice Learning: Overcoming limitations of Winner-takes-all with annealing](https://mlsanity.com/paper/2407.15580/) (2024)
- [Towards Improved Room Impulse Response Estimation for Speech Recognition](https://mlsanity.com/paper/2211.04473/) (2022)
- [ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding](https://mlsanity.com/paper/2207.09514/) (2022)
- [Diffusion-based Generative Modeling with Discriminative Guidance for Streamable Speech Enhancement](https://mlsanity.com/paper/2406.13471/) (2024)
- [The Makerere Radio Speech Corpus: A Luganda Radio Corpus for Automatic Speech Recognition](https://mlsanity.com/paper/2206.09790/) (2022)
- [Multi-Microphone Speaker Separation by Spatial Regions](https://mlsanity.com/paper/2303.07143/) (2023)
- [Exploration of Adapter for Noise Robust Automatic Speech Recognition](https://mlsanity.com/paper/2402.18275/) (2024)
- [Discovery of Single Independent Latent Variable](https://mlsanity.com/paper/2110.05887/) (2021)
- [Multiple Choice Learning for Efficient Speech Separation with Many Speakers](https://mlsanity.com/paper/2411.18497/) (2024)
- [Contextualized Token Discrimination for Speech Search Query Correction](https://mlsanity.com/paper/2509.04393/) (2025)
- [ESPnet-ST IWSLT 2021 Offline Speech Translation System](https://mlsanity.com/paper/2107.00636/) (2021)
- [TRNet: Two-level Refinement Network leveraging Speech Enhancement for Noise Robust Speech Emotion Recognition](https://mlsanity.com/paper/2404.12979/) (2024)
- [ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration](https://mlsanity.com/paper/2409.09506/) (2024)
- [Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling](https://mlsanity.com/paper/2508.06393/) (2025)
- [TISDiSS: A Training-Time and Inference-Time Scalable Framework for Discriminative Source Separation](https://mlsanity.com/paper/2509.15666/) (2025)

## Abstract (arXiv)

We present ESPnet-SE, which is designed for the quick development of speech enhancement and speech separation systems in a single framework, along with the optional downstream speech recognition module. ESPnet-SE is a new project which integrates rich automatic speech recognition related models, resources and systems to support and validate the proposed front-end implementation (i.e. speech enhancement and separation).It is capable of processing both single-channel and multi-channel data, with various functionalities including dereverberation, denoising and source separation. We provide all-in-one recipes including data pre-processing, feature extraction, training and evaluation pipelines for a wide range of benchmark datasets. This paper describes the design of the toolkit, several important functionalities, especially the speech recognition integration, which differentiates ESPnet-SE from other open source toolkits, and experimental results with major benchmark datasets.