mlsanityData to 7 Oct 2026

AudioCodecBench: A Comprehensive Benchmark for Audio Codec Evaluation

arXiv 2509.02349 ↗First posted 2 Sep 2025cs.SD cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have been widely applied in speech and music. This tendency has led to a focus on audio tokenization for Large Models (LMs). Unlike semantic-only text tokens, audio tokens must both capture global semantic content and preserve fine-grained acoustic details. Moreover, they provide a discrete method for speech and music that can be effectively integrated into MLLMs. However, existing research is unsuitable in the definitions of semantic tokens and acoustic tokens. In addition, the evaluation of different codecs typically concentrates on specific domains or tasks, such as reconstruction or Automatic Speech Recognition (ASR) task, which prevents fair and comprehensive comparisons. To address these problems, this paper provides suitable definitions for semantic and acoustic tokens and introduces a systematic evaluation framework. This framework allows for a comprehensive assessment of codecs' capabilities which evaluate across four dimensions: audio reconstruction metric, codebook index (ID) stability, decoder-only transformer perplexity, and performance on downstream probe tasks. Our results show the correctness of the provided suitable definitions and the correlation among reconstruction metrics, codebook ID stability, downstream probe tasks and perplexity.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 45 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

31 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024DASB - Discrete Audio and Speech Benchmark20 Jun

Other references · 30

  1. 2017Squeeze-and-Excitation Networkscited by 4,763
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  11. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  12. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
18 more
  1. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  2. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  3. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  4. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  5. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
  6. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  7. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  8. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
  9. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  10. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 70
  11. 2022Vocalsound: A Dataset for Improving Human Vocal Sounds Recognitioncited by 62
  12. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  13. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  14. 2024VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Musiccited by 30
  15. 2025XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecscited by 21
  16. 2025ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modelingcited by 12
  17. 2019The NIGENS General Sound Events Databasecited by 8
  18. 2024Code Drift: Towards Idempotent Neural Audio Codecscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
  2. 2025Towards Audio Token Compression in Large Audio Language Models
  3. 2025UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook
  4. 2026HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec
  5. 2025Make Some Noise: Towards LLM audio reasoning and generation using sound tokens
  6. 2025Continuous Audio Language Models
  7. 2026LoSATok: Low-Dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation
  8. 2026DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models