# ESPnet: End-to-End Speech Processing Toolkit

arXiv: https://arxiv.org/abs/1804.00015 · HTML: https://mlsanity.com/paper/1804.00015/

## Key facts

- Cited by 558 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2018.

## Built on by (most cited first, 30 of 558)

- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding](https://mlsanity.com/paper/2207.02971/) (2022)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [WaveFake: A Data Set to Facilitate Audio Deepfake Detection](https://mlsanity.com/paper/2111.02813/) (2021)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer](https://mlsanity.com/paper/2401.16658/) (2024)
- [ML-SUPERB: Multilingual Speech Universal PERformance Benchmark](https://mlsanity.com/paper/2305.10615/) (2023)
- [Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition](https://mlsanity.com/paper/2109.06870/) (2021)
- [mHuBERT-147: A Compact Multilingual HuBERT Model](https://mlsanity.com/paper/2406.06371/) (2024)
- [Earnings-21: A Practical Benchmark for ASR in the Wild](https://mlsanity.com/paper/2104.11348/) (2021)
- [Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictions](https://mlsanity.com/paper/2104.02724/) (2021)
- [Mamba in Speech: Towards an Alternative to Self-Attention](https://mlsanity.com/paper/2405.12609/) (2024)
- [YODAS: Youtube-Oriented Dataset for Audio and Speech](https://mlsanity.com/paper/2406.00899/) (2024)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [Can Generative Large Language Models Perform ASR Error Correction?](https://mlsanity.com/paper/2307.04172/) (2023)

## Abstract (arXiv)

This paper introduces a new open source platform for end-to-end speech processing named ESPnet. ESPnet mainly focuses on end-to-end automatic speech recognition (ASR), and adopts widely-used dynamic neural network toolkits, Chainer and PyTorch, as a main deep learning engine. ESPnet also follows the Kaldi ASR toolkit style for data processing, feature extraction/format, and recipes to provide a complete setup for speech recognition and other speech processing experiments. This paper explains a major architecture of this software platform, several important functionalities, which differentiate ESPnet from other open source ASR toolkits, and experimental results with major ASR benchmarks.