mlsanityData to 7 Oct 2026

Scaling Speech Tokenizers with Diffusion Autoencoders

arXiv 2602.06602 ↗First posted 6 Feb 2026cs.SD cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Speech tokenizers are foundational to speech language models, yet existing approaches face two major challenges: (1) balancing trade-offs between encoding semantics for understanding and acoustics for reconstruction, and (2) achieving low bit rates and low token rates. We propose Speech Diffusion Tokenizer (SiTok), a diffusion autoencoder that jointly learns semantic-rich representations through supervised learning and enables high-fidelity audio reconstruction with diffusion. We scale SiTok to 1.6B parameters and train it on 2 million hours of speech. Experiments show that SiTok outperforms strong baselines on understanding, reconstruction and generation tasks, at an extremely low token rate of 12.5 Hz and a bit-rate of 200 bits-per-second.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 72 of its 80 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

72 of 80 references.

References · 72

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
38 more
  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2019Root Mean Square Layer Normalizationcited by 1,425
  3. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  5. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  6. 2025Qwen2.5-Omni Technical Reportcited by 704
  7. 2022High Fidelity Neural Audio Compressioncited by 673
  8. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  9. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  11. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  12. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  13. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  14. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  15. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  16. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  17. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  18. 2024One Step Diffusion via Shortcut Modelscited by 373
  19. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  20. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  21. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  22. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  23. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  24. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  25. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
  26. 2025Kimi-Audio Technical Reportcited by 210
  27. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
  28. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  29. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  30. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  31. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  32. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  33. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  34. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  35. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 110
  36. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  37. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  38. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026End-to-End Training for Discrete Token LLM based TTS Systemcited by 0
  2. 2026Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokenscited by 0