mlsanityData to 14 Oct 2026

ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech

arXiv 2409.15897 ↗First posted 24 Sep 2024eess.AS cs.SDcited by 26

Not on the map. Why?

Abstract (arXiv)

Neural codecs have become crucial to recent speech and audio generation research. In addition to signal compression capabilities, discrete codecs have also been found to enhance downstream training efficiency and compatibility with autoregressive language models. However, as extensive downstream applications are investigated, challenges have arisen in ensuring fair comparisons across diverse applications. To address these issues, we present a new open-source platform ESPnet-Codec, which is built on ESPnet and focuses on neural codec training and evaluation. ESPnet-Codec offers various recipes in audio, music, and speech for training and evaluation using several widely adopted codec models. Together with ESPnet-Codec, we present VERSA, a standalone evaluation toolkit, which provides a comprehensive evaluation of codec performance over 20 audio evaluation metrics. Notably, we demonstrate that ESPnet-Codec can be integrated into six ESPnet tasks, supporting diverse applications.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Sep 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 12.

26 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  3. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  4. 2025ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systemscited by 7
  5. 2025SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessmentcited by 7
  6. 2025Chain-of-Thought Training for Open E2E Spoken Dialogue Systemscited by 5
  7. 2025Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systemscited by 5
  8. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  9. 2025On-device Streaming Discrete Speech Unitscited by 4
  10. 2026Bagpiper: Solving Open-Ended Audio Tasks via Rich Captionscited by 4
  11. 2025Towards Efficient Speech-Text Jointly Decoding within One Speech Language Modelcited by 1
  12. 2025Factorized RVQ-GAN For Disentangled Speech Tokenizationcited by 1
14 more
  1. 2025Benchmarking Prosody Encoding in Discrete Speech Tokenscited by 1
  2. 2025Training Flow Matching Models with Reliable Labels via Self-Purificationcited by 1
  3. 2025Do Foundational Audio Encoders Understand Music Structure?cited by 1
  4. 2026Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Taskscited by 1
  5. 2025Task-Specific Audio Coding for Machines: Machine-Learned Latent Features Are Codes for That Machinecited by 0
  6. 2026LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancementcited by 0
  7. 2026Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbationscited by 0
  8. 2026PASQA: Pitch-Accent-Focused Speech Quality Assessment Model Trained on Synthetic Speech with Accent Errorscited by 0
  9. 2026An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generationcited by 0
  10. 2026Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokenscited by 0
  11. 2026LACE: Layer-Wise Compression for Dynamic Frame Rate Codecscited by 0
  12. 2026HaikuS2S: A Cascaded System For Responding In Versecited by 0
  13. 2026YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speechcited by 0
  14. 2026How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Qualitycited by 0