# Codec-SUPERB: An In-Depth Analysis of Sound Codec Models

arXiv: https://arxiv.org/abs/2402.13071 · HTML: https://mlsanity.com/paper/2402.13071/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2024.

## Built on by (most cited first, 26 of 26)

- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [M-Prometheus: A Suite of Open Multilingual LLM Judges](https://mlsanity.com/paper/2504.04953/) (2025)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [ViSAGe: Video-to-Spatial Audio Generation](https://mlsanity.com/paper/2506.12199/) (2025)
- [Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model](https://mlsanity.com/paper/2506.08967/) (2025)
- [Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative](https://mlsanity.com/paper/2508.09294/) (2025)
- [Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate](https://mlsanity.com/paper/2505.16845/) (2025)
- [Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy](https://mlsanity.com/paper/2506.22023/) (2025)
- [Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoders](https://mlsanity.com/paper/2507.07867/) (2025)
- [Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering](https://mlsanity.com/paper/2606.11386/) (2026)
- [EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance](https://mlsanity.com/paper/2409.01201/) (2024)
- [Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs](https://mlsanity.com/paper/2511.16639/) (2025)
- [Analysing the Language of Neural Audio Codecs](https://mlsanity.com/paper/2509.01390/) (2025)
- [Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec](https://mlsanity.com/paper/2603.05887/) (2026)
- [Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning](https://mlsanity.com/paper/2409.01160/) (2024)
- [Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model](https://mlsanity.com/paper/2506.04518/) (2025)
- [Benchmarking Prosody Encoding in Discrete Speech Tokens](https://mlsanity.com/paper/2508.11224/) (2025)
- [Pseudo2Real: Task Arithmetic for Pseudo-Label Correction in Automatic Speech Recognition](https://mlsanity.com/paper/2510.08047/) (2025)
- [TiCo: Time-Controllable Spoken Dialogue Model](https://mlsanity.com/paper/2603.22267/) (2026)
- [Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions](https://mlsanity.com/paper/2608.31037/) (2026)
- [Towards Generalized Source Tracing for Codec-Based Deepfake Speech](https://mlsanity.com/paper/2506.07294/) (2025)
- [Task-Specific Audio Coding for Machines: Machine-Learned Latent Features Are Codes for That Machine](https://mlsanity.com/paper/2507.12701/) (2025)
- [FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs](https://mlsanity.com/paper/2509.11425/) (2025)
- [Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment](https://mlsanity.com/paper/2606.12940/) (2026)
- [GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding](https://mlsanity.com/paper/2610.04651/) (2026)

## Abstract (arXiv)

The sound codec's dual roles in minimizing data transmission latency and serving as tokenizers underscore its critical importance. Recent years have witnessed significant developments in codec models. The ideal sound codec should preserve content, paralinguistics, speakers, and audio information. However, the question of which codec achieves optimal sound information preservation remains unanswered, as in different papers, models are evaluated on their selected experimental settings. This study introduces Codec-SUPERB, an acronym for Codec sound processing Universal PERformance Benchmark. It is an ecosystem designed to assess codec models across representative sound applications and signal-level metrics rooted in sound domain knowledge.Codec-SUPERB simplifies result sharing through an online leaderboard, promoting collaboration within a community-driven benchmark database, thereby stimulating new development cycles for codecs. Furthermore, we undertake an in-depth analysis to offer insights into codec models from both application and signal perspectives, diverging from previous codec papers mainly concentrating on signal-level comparisons. Finally, we will release codes, the leaderboard, and data to accelerate progress within the community.