# DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation

arXiv: https://arxiv.org/abs/2505.13000 · HTML: https://mlsanity.com/paper/2505.13000/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2025.

## Built on by (most cited first, 21 of 21)

- [SpeechJudge: Towards Human-Level Judgment for Speech Naturalness](https://mlsanity.com/paper/2511.07931/) (2025)
- [MOSS-TTS Technical Report](https://mlsanity.com/paper/2603.18090/) (2026)
- [TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling](https://mlsanity.com/paper/2508.16790/) (2025)
- [MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation](https://mlsanity.com/paper/2506.00385/) (2025)
- [Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization](https://mlsanity.com/paper/2601.23174/) (2026)
- [JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention](https://mlsanity.com/paper/2512.07168/) (2025)
- [Scaling Speech Tokenizers with Diffusion Autoencoders](https://mlsanity.com/paper/2602.06602/) (2026)
- [LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization](https://mlsanity.com/paper/2506.16738/) (2025)
- [U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation](https://mlsanity.com/paper/2510.16718/) (2025)
- [DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion](https://mlsanity.com/paper/2601.09239/) (2026)
- [TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context](https://mlsanity.com/paper/2609.08703/) (2026)
- [StepAudio 3 Music Technical Report](https://mlsanity.com/paper/2609.16034/) (2026)
- [Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditions](https://mlsanity.com/paper/2509.24457/) (2025)
- [S-PRESSO: Ultra Low Bitrate Sound Effect Compression With Diffusion Autoencoders And Offline Quantization](https://mlsanity.com/paper/2602.15082/) (2026)
- [PoDAR: Power-Disentangled Audio Representation for Generative Modeling](https://mlsanity.com/paper/2605.10084/) (2026)
- [CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding](https://mlsanity.com/paper/2606.04418/) (2026)
- [HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec](https://mlsanity.com/paper/2606.06743/) (2026)
- [Content is What Remains: Invariant Speech Tokenization from Parallel Utterances](https://mlsanity.com/paper/2607.19033/) (2026)
- [TokenMapper: A Step Toward Interoperable Speech Token Translation](https://mlsanity.com/paper/2609.12563/) (2026)
- [Q-SPT: Learnable Query-Based Compression for Low-Frame-Rate Speech Tokenization](https://mlsanity.com/paper/2610.01492/) (2026)
- [GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding](https://mlsanity.com/paper/2610.04651/) (2026)

## Abstract (arXiv)

Neural audio codecs form the foundational building blocks for language model (LM)-based speech generation. Typically, there is a trade-off between frame rate and audio quality. This study introduces a low-frame-rate, semantically enhanced codec model. Existing approaches distill semantically rich self-supervised (SSL) representations into the first-layer codec tokens. This work proposes DualCodec, a dual-stream encoding approach that integrates SSL and waveform representations within an end-to-end codec framework. In this setting, DualCodec enhances the semantic information in the first-layer codec and enables the codec system to maintain high audio quality while operating at a low frame rate. Note that a low-frame-rate codec improves the efficiency of speech generation. Experimental results on audio codec and speech generation tasks confirm the effectiveness of the proposed DualCodec compared to state-of-the-art codec systems, such as Mimi Codec, SpeechTokenizer, DAC, and Encodec. Demos are available at: https://dualcodec.github.io, code is available at: https://github.com/jiaqili3/DualCodec