# The 2020 ESPnet update: new features, broadened applications, performance improvements, and future plans

arXiv: https://arxiv.org/abs/2012.13006 · HTML: https://mlsanity.com/paper/2012.13006/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2020.

## Built on by (most cited first, 14 of 14)

- [SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation](https://mlsanity.com/paper/2205.08180/) (2022)
- [A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation](https://mlsanity.com/paper/2110.05249/) (2021)
- [Anonymizing Speech with Generative Adversarial Networks to Preserve Speaker Privacy](https://mlsanity.com/paper/2210.07002/) (2022)
- [Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction](https://mlsanity.com/paper/2110.15430/) (2021)
- [Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0](https://mlsanity.com/paper/2110.03560/) (2021)
- [Towards Identity Preserving Normal to Dysarthric Voice Conversion](https://mlsanity.com/paper/2110.08213/) (2021)
- [Speaker Anonymization with Phonetic Intermediate Representations](https://mlsanity.com/paper/2207.04834/) (2022)
- [A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion](https://mlsanity.com/paper/2106.01415/) (2021)
- [A Conformer Based Acoustic Model for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2203.00725/) (2022)
- [Integrating Lattice-Free MMI into End-to-End Speech Recognition](https://mlsanity.com/paper/2203.15614/) (2022)
- [Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training](https://mlsanity.com/paper/2404.10922/) (2024)
- [Dynamic Acoustic Unit Augmentation With BPE-Dropout for Low-Resource End-to-End Speech Recognition](https://mlsanity.com/paper/2103.07186/) (2021)
- [Dynamic Alignment Mask CTC: Improved Mask-CTC with Aligned Cross Entropy](https://mlsanity.com/paper/2303.07687/) (2023)
- [JoeyS2T: Minimalistic Speech-to-Text Modeling with JoeyNMT](https://mlsanity.com/paper/2210.02545/) (2022)

## Abstract (arXiv)

This paper describes the recent development of ESPnet (https://github.com/espnet/espnet), an end-to-end speech processing toolkit. This project was initiated in December 2017 to mainly deal with end-to-end speech recognition experiments based on sequence-to-sequence modeling. The project has grown rapidly and now covers a wide range of speech processing applications. Now ESPnet also includes text to speech (TTS), voice conversation (VC), speech translation (ST), and speech enhancement (SE) with support for beamforming, speech separation, denoising, and dereverberation. All applications are trained in an end-to-end manner, thanks to the generic sequence to sequence modeling properties, and they can be further integrated and jointly optimized. Also, ESPnet provides reproducible all-in-one recipes for these applications with state-of-the-art performance in various benchmarks by incorporating transformer, advanced data augmentation, and conformer. This project aims to provide up-to-date speech processing experience to the community so that researchers in academia and various industry scales can develop their technologies collaboratively.