# Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation

arXiv: https://arxiv.org/abs/2406.07422 · HTML: https://mlsanity.com/paper/2406.07422/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2024.

## Built on by (most cited first, 23 of 23)

- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [WavChat: A Survey of Spoken Dialogue Models](https://mlsanity.com/paper/2411.13577/) (2024)
- [Scaling Transformers for Low-Bitrate High-Quality Speech Coding](https://mlsanity.com/paper/2411.19842/) (2024)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [M-Prometheus: A Suite of Open Multilingual LLM Judges](https://mlsanity.com/paper/2504.04953/) (2025)
- [FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks](https://mlsanity.com/paper/2502.04465/) (2025)
- [Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding](https://mlsanity.com/paper/2410.21951/) (2024)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec](https://mlsanity.com/paper/2410.15764/) (2024)
- [KALL-E:Autoregressive Speech Synthesis with Next-Distribution Prediction](https://mlsanity.com/paper/2412.16846/) (2024)
- [Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate](https://mlsanity.com/paper/2505.16845/) (2025)
- [SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancement](https://mlsanity.com/paper/2509.24708/) (2025)
- [The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge](https://mlsanity.com/paper/2410.23815/) (2024)
- [UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information](https://mlsanity.com/paper/2505.17426/) (2025)
- [HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling](https://mlsanity.com/paper/2507.18897/) (2025)
- [The ISCSLP 2024 Conversational Voice Clone (CoVoC) Challenge: Tasks, Results and Findings](https://mlsanity.com/paper/2411.00064/) (2024)
- [LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization](https://mlsanity.com/paper/2506.16738/) (2025)
- [Speech Tokenizer is Key to Consistent Representation](https://mlsanity.com/paper/2507.06802/) (2025)
- [Two-Dimensional Quantization for Geometry-Aware Audio Coding](https://mlsanity.com/paper/2512.01537/) (2025)
- [CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding](https://mlsanity.com/paper/2606.04418/) (2026)
- [Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment](https://mlsanity.com/paper/2606.12940/) (2026)
- [TokenMapper: A Step Toward Interoperable Speech Token Translation](https://mlsanity.com/paper/2609.12563/) (2026)

## Abstract (arXiv)

The multi-codebook speech codec enables the application of large language models (LLM) in TTS but bottlenecks efficiency and robustness due to multi-sequence prediction. To avoid this obstacle, we propose Single-Codec, a single-codebook single-sequence codec, which employs a disentangled VQ-VAE to decouple speech into a time-invariant embedding and a phonetically-rich discrete sequence. Furthermore, the encoder is enhanced with 1) contextual modeling with a BLSTM module to exploit the temporal information, 2) a hybrid sampling module to alleviate distortion from upsampling and downsampling, and 3) a resampling module to encourage discrete units to carry more phonetic information. Compared with multi-codebook codecs, e.g., EnCodec and TiCodec, Single-Codec demonstrates higher reconstruction quality with a lower bandwidth of only 304bps. The effectiveness of Single-Code is further validated by LLM-TTS experiments, showing improved naturalness and intelligibility.