# Recent Advances in Discrete Speech Tokens: A Review

arXiv: https://arxiv.org/abs/2502.06490 · HTML: https://mlsanity.com/paper/2502.06490/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 244 of its 292 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 30 of 244)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)

## Built on by (most cited first, 30 of 43)

- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data](https://mlsanity.com/paper/2505.20166/) (2025)
- [AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering](https://mlsanity.com/paper/2601.12248/) (2026)
- [How Contrastive Decoding Enhances Large Audio Language Models](https://mlsanity.com/paper/2603.09232/) (2026)
- [FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speech](https://mlsanity.com/paper/2606.23190/) (2026)
- [Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy](https://mlsanity.com/paper/2506.22023/) (2025)
- [ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models](https://mlsanity.com/paper/2507.20091/) (2025)
- [Direct Preference Optimization for Speech Autoregressive Diffusion Models](https://mlsanity.com/paper/2509.18928/) (2025)
- [StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs](https://mlsanity.com/paper/2509.22220/) (2025)
- [Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization](https://mlsanity.com/paper/2601.23174/) (2026)
- [Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models](https://mlsanity.com/paper/2604.25591/) (2026)
- [Analysing the Language of Neural Audio Codecs](https://mlsanity.com/paper/2509.01390/) (2025)
- [Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates](https://mlsanity.com/paper/2509.09550/) (2025)
- [Beyond WER: Probing Whisper's Sub-token Decoder Across Diverse Language Resource Levels](https://mlsanity.com/paper/2509.25516/) (2025)
- [Rethinking Discrete Speech Representation Tokens for Accent Generation](https://mlsanity.com/paper/2601.19786/) (2026)
- [Sylber 2.0: A Universal Syllable Embedding](https://mlsanity.com/paper/2601.22306/) (2026)
- [Scaling Speech Tokenizers with Diffusion Autoencoders](https://mlsanity.com/paper/2602.06602/) (2026)
- [HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding](https://mlsanity.com/paper/2605.29948/) (2026)
- [Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model](https://mlsanity.com/paper/2506.04518/) (2025)
- [Benchmarking Prosody Encoding in Discrete Speech Tokens](https://mlsanity.com/paper/2508.11224/) (2025)
- [DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion](https://mlsanity.com/paper/2601.09239/) (2026)
- [VocBulwark: Towards Practical Generative Speech Watermarking via Additional-Parameter Injection](https://mlsanity.com/paper/2601.22556/) (2026)
- [The Equalizer: Introducing Shape-Gain Decomposition in Neural Audio Codecs](https://mlsanity.com/paper/2602.15491/) (2026)
- [WavSLM: Single-Stream Speech Language Modeling via WavLM Distillation](https://mlsanity.com/paper/2603.05299/) (2026)
- [TiCo: Time-Controllable Spoken Dialogue Model](https://mlsanity.com/paper/2603.22267/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning](https://mlsanity.com/paper/2606.04939/) (2026)
- [The Sound of Absence: Audio-Language Embedding Models Struggle with Negation](https://mlsanity.com/paper/2607.12290/) (2026)
- [Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions](https://mlsanity.com/paper/2608.31037/) (2026)

## Abstract (arXiv)

The rapid advancement of speech generation technologies in the era of large language models (LLMs) has established discrete speech tokens as a foundational paradigm for speech representation. These tokens, characterized by their discrete, compact, and concise nature, are not only advantageous for efficient transmission and storage, but also inherently compatible with the language modeling framework, enabling seamless integration of speech into text-dominated LLM architectures. Current research categorizes discrete speech tokens into two principal classes: acoustic tokens and semantic tokens, each of which has evolved into a rich research domain characterized by unique design philosophies and methodological approaches. This survey systematically synthesizes the existing taxonomy and recent innovations in discrete speech tokenization, conducts a critical examination of the strengths and limitations of each paradigm, and presents systematic experimental comparisons across token types. Furthermore, we identify persistent challenges in the field and propose potential research directions, aiming to offer actionable insights to inspire future advancements in the development and application of discrete speech tokens.