mlsanityData to 14 Oct 2026

Recent Advances in Discrete Speech Tokens: A Review

arXiv 2502.06490 ↗First posted 10 Feb 2025eess.AS cs.AI cs.MMcited by 43

Not on the map. Why?

Abstract (arXiv)

The rapid advancement of speech generation technologies in the era of large language models (LLMs) has established discrete speech tokens as a foundational paradigm for speech representation. These tokens, characterized by their discrete, compact, and concise nature, are not only advantageous for efficient transmission and storage, but also inherently compatible with the language modeling framework, enabling seamless integration of speech into text-dominated LLM architectures. Current research categorizes discrete speech tokens into two principal classes: acoustic tokens and semantic tokens, each of which has evolved into a rich research domain characterized by unique design philosophies and methodological approaches. This survey systematically synthesizes the existing taxonomy and recent innovations in discrete speech tokenization, conducts a critical examination of the strengths and limitations of each paradigm, and presents systematic experimental comparisons across token types. Furthermore, we identify persistent challenges in the field and propose potential research directions, aiming to offer actionable insights to inspire future advancements in the development and application of discrete speech tokens.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 244 of its 292 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

244 of 292 references.

References · 244

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2022Flow Matching for Generative Modelingcited by 5,702
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
38 more
  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  3. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  11. 2022High Fidelity Neural Audio Compressioncited by 687
  12. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  13. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  14. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  15. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  16. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  17. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  18. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  19. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  20. 2020Jukebox: A Generative Model for Musiccited by 542
  21. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  22. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  23. 2023Simple and Controllable Music Generationcited by 488
  24. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  25. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  26. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  27. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  28. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  29. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  30. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  31. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  32. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  33. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  34. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  35. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  36. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  37. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  38. 2022Masked Autoencoders that Listencited by 273

50 most cited of 244 All 244 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 26.

43 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  3. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  4. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  5. 2026How Contrastive Decoding Enhances Large Audio Language Modelscited by 6
  6. 2026FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speechcited by 5
  7. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  8. 2025ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Modelscited by 4
  9. 2025Direct Preference Optimization for Speech Autoregressive Diffusion Modelscited by 4
  10. 2025StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMscited by 4
  11. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 4
  12. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
31 more
  1. 2025Analysing the Language of Neural Audio Codecscited by 2
  2. 2025Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-ratescited by 2
  3. 2025Beyond WER: Probing Whisper's Sub-token Decoder Across Diverse Language Resource Levelscited by 2
  4. 2026Rethinking Discrete Speech Representation Tokens for Accent Generationcited by 2
  5. 2026Sylber 2.0: A Universal Syllable Embeddingcited by 2
  6. 2026Scaling Speech Tokenizers with Diffusion Autoencoderscited by 2
  7. 2026HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understandingcited by 2
  8. 2025Towards Efficient Speech-Text Jointly Decoding within One Speech Language Modelcited by 1
  9. 2025Benchmarking Prosody Encoding in Discrete Speech Tokenscited by 1
  10. 2026DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusioncited by 1
  11. 2026VocBulwark: Towards Practical Generative Speech Watermarking via Additional-Parameter Injectioncited by 1
  12. 2026The Equalizer: Introducing Shape-Gain Decomposition in Neural Audio Codecscited by 1
  13. 2026WavSLM: Single-Stream Speech Language Modeling via WavLM Distillationcited by 1
  14. 2026TiCo: Time-Controllable Spoken Dialogue Modelcited by 1
  15. 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documentscited by 1
  16. 2026UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioningcited by 1
  17. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  18. 2026Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditionscited by 1
  19. 2025Towards Generalized Source Tracing for Codec-Based Deepfake Speechcited by 0
  20. 2025Task-Specific Audio Coding for Machines: Machine-Learned Latent Features Are Codes for That Machinecited by 0
  21. 2025TokenChain: A Discrete Speech Chain via Semantic Token Modelingcited by 0
  22. 2025The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASRcited by 0
  23. 2025Unsupervised Discovery of Long-Term Spatiotemporal Periodic Workflows in Human Activitiescited by 0
  24. 2026Decoding Order Matters in Autoregressive Speech Synthesiscited by 0
  25. 2026Generalized Radius and Integrated Codebook Transforms for Differentiable Vector Quantizationcited by 0
  26. 2026Thinking-while-speaking: A Controlled, Interleaved Reasoning Method for Real-Time Speech Generationcited by 0
  27. 2026Mask, Sample, Revise: A Revisable CTMC Inference Stack for Guided Discrete Flow Matching Text-to-Speechcited by 0
  28. 2026An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generationcited by 0
  29. 2026Shifting Relational Paradigms for Affective Computing: Affective Resonance, Vitality Affects, and Vocal Interaction Fieldscited by 0
  30. 2026Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMscited by 0
  31. 2026Controlling Speaking Rate in Autoregressive TTS via Activation Steeringcited by 0