mlsanityData to 14 Oct 2026

SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models

arXiv 2308.16692 ↗First posted 31 Aug 2023cs.CL cs.SD eess.AScited by 87

Not on the map. Why?

Abstract (arXiv)

Current speech large language models build upon discrete speech representations, which can be categorized into semantic tokens and acoustic tokens. However, existing speech tokens are not specifically designed for speech language modeling. To assess the suitability of speech tokens for building speech language models, we established the first benchmark, SLMTokBench. Our results indicate that neither semantic nor acoustic tokens are ideal for this purpose. Therefore, we propose SpeechTokenizer, a unified speech tokenizer for speech large language models. SpeechTokenizer adopts the Encoder-Decoder architecture with residual vector quantization (RVQ). Unifying semantic and acoustic tokens, SpeechTokenizer disentangles different aspects of speech information hierarchically across different RVQ layers. Furthermore, We construct a Unified Speech Language Model (USLM) leveraging SpeechTokenizer. Experiments show that SpeechTokenizer performs comparably to EnCodec in speech reconstruction and demonstrates strong performance on the SLMTokBench benchmark. Also, USLM outperforms VALL-E in zero-shot Text-to-Speech tasks. Code and models are available at https://github.com/ZhangXInFD/SpeechTokenizer/.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

30 of 37 references.

References · 30

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2016Layer Normalizationcited by 5,669
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  9. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  11. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  12. 2022High Fidelity Neural Audio Compressioncited by 687
18 more
  1. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  2. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  3. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  4. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  6. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  7. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  8. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  9. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  10. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 235
  11. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
  12. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  13. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  14. 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
  15. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 97
  16. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 88
  17. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  18. 2023UnifySpeech: A Unified Framework for Zero-shot Text-to-Speech and Voice Conversioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 32; 2025: 29; 2026: 25.

87 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  3. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  4. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  5. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  6. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  7. 2026Qwen3-TTS Technical Reportcited by 118
  8. 2025Step-Audio 2 Technical Reportcited by 105
  9. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  10. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  11. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  12. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
38 more
  1. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 65
  2. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  3. 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 47
  4. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  5. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  6. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  7. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  8. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  9. 2025XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecscited by 23
  10. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenizationcited by 22
  11. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
  12. 2025Fun-Audio-Chat Technical Reportcited by 19
  13. 2024Coding Speech through Vocal Tract Kinematicscited by 18
  14. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  15. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  16. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  17. 2026MOSS-TTS Technical Reportcited by 14
  18. 2026GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Taskscited by 14
  19. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  20. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 13
  21. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 11
  22. 2024dMel: Speech Tokenization made Simplecited by 11
  23. 2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inferencecited by 11
  24. 2026MOSS-Audio Technical Reportcited by 11
  25. 2025Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Playcited by 10
  26. 2024LAST: Language Model Aware Speech Tokenizationcited by 9
  27. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillationcited by 9
  28. 2024Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representationscited by 8
  29. 2025InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compressioncited by 8
  30. 2025MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generationcited by 7
  31. 2024Text-To-Speech Synthesis In The Wildcited by 6
  32. 2024Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistantcited by 6
  33. 2024Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibilitycited by 5
  34. 2024DM-Codec: Distilling Multimodal Representations for Speech Tokenizationcited by 5
  35. 2025UniWav: Towards Unified Pre-training for Speech Representation Learning and Generationcited by 5
  36. 2025SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codeccited by 5
  37. 2025ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Modelscited by 4
  38. 2025StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMscited by 4

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023RepCodec: A Speech Representation Codec for Speech Tokenization
  2. 2025Discrete Audio Tokens: More Than a Survey!
  3. 2025Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
  4. 2026Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models
  5. 2026DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion
  6. 2026UniWeTok: An Unified Binary Tokenizer with Codebook Size mathit2128 for Unified Multimodal Large Language Model
  7. 2025LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization
  8. 2025Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations