# ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech

arXiv: https://arxiv.org/abs/2409.15897 · HTML: https://mlsanity.com/paper/2409.15897/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2024.

## Built on by (most cited first, 26 of 26)

- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [ESPnet-SpeechLM: An Open Speech Language Model Toolkit](https://mlsanity.com/paper/2502.15218/) (2025)
- [ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems](https://mlsanity.com/paper/2503.08533/) (2025)
- [SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessment](https://mlsanity.com/paper/2510.01812/) (2025)
- [Chain-of-Thought Training for Open E2E Spoken Dialogue Systems](https://mlsanity.com/paper/2506.00722/) (2025)
- [Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systems](https://mlsanity.com/paper/2510.02066/) (2025)
- [Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2601.19063/) (2026)
- [On-device Streaming Discrete Speech Units](https://mlsanity.com/paper/2506.01845/) (2025)
- [Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions](https://mlsanity.com/paper/2602.05220/) (2026)
- [Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model](https://mlsanity.com/paper/2506.04518/) (2025)
- [Factorized RVQ-GAN For Disentangled Speech Tokenization](https://mlsanity.com/paper/2506.15456/) (2025)
- [Benchmarking Prosody Encoding in Discrete Speech Tokens](https://mlsanity.com/paper/2508.11224/) (2025)
- [Training Flow Matching Models with Reliable Labels via Self-Purification](https://mlsanity.com/paper/2509.19091/) (2025)
- [Do Foundational Audio Encoders Understand Music Structure?](https://mlsanity.com/paper/2512.17209/) (2025)
- [Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks](https://mlsanity.com/paper/2601.12205/) (2026)
- [Task-Specific Audio Coding for Machines: Machine-Learned Latent Features Are Codes for That Machine](https://mlsanity.com/paper/2507.12701/) (2025)
- [LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2603.13952/) (2026)
- [Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations](https://mlsanity.com/paper/2606.19951/) (2026)
- [PASQA: Pitch-Accent-Focused Speech Quality Assessment Model Trained on Synthetic Speech with Accent Errors](https://mlsanity.com/paper/2606.20137/) (2026)
- [An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation](https://mlsanity.com/paper/2607.02119/) (2026)
- [Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens](https://mlsanity.com/paper/2607.26350/) (2026)
- [LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs](https://mlsanity.com/paper/2609.17509/) (2026)
- [HaikuS2S: A Cascaded System For Responding In Verse](https://mlsanity.com/paper/2609.23951/) (2026)
- [YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speech](https://mlsanity.com/paper/2609.29448/) (2026)
- [How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Quality](https://mlsanity.com/paper/2610.00154/) (2026)

## Abstract (arXiv)

Neural codecs have become crucial to recent speech and audio generation research. In addition to signal compression capabilities, discrete codecs have also been found to enhance downstream training efficiency and compatibility with autoregressive language models. However, as extensive downstream applications are investigated, challenges have arisen in ensuring fair comparisons across diverse applications. To address these issues, we present a new open-source platform ESPnet-Codec, which is built on ESPnet and focuses on neural codec training and evaluation. ESPnet-Codec offers various recipes in audio, music, and speech for training and evaluation using several widely adopted codec models. Together with ESPnet-Codec, we present VERSA, a standalone evaluation toolkit, which provides a comprehensive evaluation of codec performance over 20 audio evaluation metrics. Notably, we demonstrate that ESPnet-Codec can be integrated into six ESPnet tasks, supporting diverse applications.