# Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generation

arXiv: https://arxiv.org/abs/2505.24496 · HTML: https://mlsanity.com/paper/2505.24496/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2025.

## Built on by (most cited first, 6 of 6)

- [Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap](https://mlsanity.com/paper/2603.01502/) (2026)
- [VoxServe: Streaming-Centric Serving System for Speech Language Models](https://mlsanity.com/paper/2602.00269/) (2026)
- [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026)
- [Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTS](https://mlsanity.com/paper/2605.30748/) (2026)
- [A Simple Method to Enhance Pre-trained Language Models with Speech Tokens for Classification](https://mlsanity.com/paper/2512.07571/) (2025)
- [SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Models](https://mlsanity.com/paper/2603.09215/) (2026)

## Abstract (arXiv)

Neural audio codecs, used as speech tokenizers, have demonstrated remarkable potential in the field of speech generation. However, to ensure high-fidelity audio reconstruction, neural audio codecs typically encode audio into long sequences of speech tokens, posing a significant challenge for downstream language models in long-context modeling. We observe that speech token sequences exhibit short-range dependency: due to the monotonic alignment between text and speech in text-to-speech (TTS) tasks, the prediction of the current token primarily relies on its local context, while long-range tokens contribute less to the current token prediction and often contain redundant information. Inspired by this observation, we propose a compressed-to-fine language modeling approach to address the challenge of long sequence speech tokens within neural codec language models: (1) Fine-grained Initial and Short-range Information: Our approach retains the prompt and local tokens during prediction to ensure text alignment and the integrity of paralinguistic information; (2) Compressed Long-range Context: Our approach compresses long-range token spans into compact representations to reduce redundant information while preserving essential semantics. Extensive experiments on various neural audio codecs and downstream language models validate the effectiveness and generalizability of the proposed approach, highlighting the importance of token compression in improving speech generation within neural codec language models. The demo of audio samples will be available at https://anonymous.4open.science/r/SpeechTokenPredictionViaCompressedToFinedLM.