# UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook

arXiv: https://arxiv.org/abs/2502.20067 · HTML: https://mlsanity.com/paper/2502.20067/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2025.

## Built on by (most cited first, 10 of 10)

- [M-Prometheus: A Suite of Open Multilingual LLM Judges](https://mlsanity.com/paper/2504.04953/) (2025)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [Token-Based Audio Inpainting via Discrete Diffusion](https://mlsanity.com/paper/2507.08333/) (2025)
- [UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstruction](https://mlsanity.com/paper/2601.02776/) (2026)
- [Scaling Speech Tokenizers with Diffusion Autoencoders](https://mlsanity.com/paper/2602.06602/) (2026)
- [Alethia: A Foundational Encoder for Voice Deepfakes](https://mlsanity.com/paper/2605.00251/) (2026)
- [AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation](https://mlsanity.com/paper/2606.30811/) (2026)
- [Improving X-Codec-2.0 for Multi-Lingual Speech: 25 Hz Latent Rate and 24 kHz Sampling](https://mlsanity.com/paper/2601.20185/) (2026)
- [SqueezeComposer: Temporal Speed-up is A Simple Trick for Long-form Music Composing](https://mlsanity.com/paper/2603.21073/) (2026)
- [How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Quality](https://mlsanity.com/paper/2610.00154/) (2026)

## Abstract (arXiv)

The emergence of audio language models is empowered by neural audio codecs, which establish critical mappings between continuous waveforms and discrete tokens compatible with language model paradigms. The evolutionary trends from multi-layer residual vector quantizer to single-layer quantizer are beneficial for language-autoregressive decoding. However, the capability to handle multi-domain audio signals through a single codebook remains constrained by inter-domain distribution discrepancies. In this work, we introduce UniCodec, a unified audio codec with a single codebook to support multi-domain audio data, including speech, music, and sound. To achieve this, we propose a partitioned domain-adaptive codebook method and domain Mixture-of-Experts strategy to capture the distinct characteristics of each audio domain. Furthermore, to enrich the semantic density of the codec without auxiliary modules, we propose a self-supervised mask prediction modeling approach. Comprehensive objective and subjective evaluations demonstrate that UniCodec achieves excellent audio reconstruction performance across the three audio domains, outperforming existing unified neural codecs with a single codebook, and even surpasses state-of-the-art domain-specific codecs on both acoustic and semantic representation capabilities.