mlsanityData to 14 Oct 2026

Optimality of FSQ Tokens for Continuous Diffusion for Categorical Data with Application to Text-to-Speech

arXiv 2606.09962 ↗First posted 8 Jun 2026cs.LG cs.AI cs.SDcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Continuous diffusion for categorical data is a framework belonging to the diffusion family and aiming at generating discrete data. The scientific interest to such models has been constantly increasing these days because researchers try to achieve a challenging goal of finding reasonable alternatives to autoregressive large language models. In this paper, we study the properties of the structure of the latent space corresponding to discrete tokens expressed in terms of Kullback-Leibler divergence on diffusion path measures and accuracy of the correct token prediction by the optimally trained diffusion model. We find that FSQ tokenization scheme has the latent space structure with the properties that make it best suited for continuous diffusion for categorical data as verified through rigorous theoretical analysis and numerical experiments. To validate our findings in real-life scenario, we train several text-to-speech diffusion models having speech tokens as intermediate acoustic features, and show that the one based on FSQ tokens indeed performs the best, and, moreover, it outperforms its strong LLM-based counterpart, at the same time being significantly smaller and faster.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 51 of its 58 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

51 of 58 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Smoothie: Smoothing Diffusion on Token Embeddings for Text Generation24 May
  2. 2025Large Language Diffusion Models14 Feb
  3. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  4. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  5. 2023Likelihood-Based Diffusion Language Models30 May
  6. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusion15 May
  7. 2022Latent Diffusion for Language Generation19 Dec
  8. 2022Empowering Diffusion Models on the Embedding Space for Text Generation19 Dec

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2022Flow Matching for Generative Modelingcited by 5,702
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  12. 2021Variational Diffusion Modelscited by 1,153
31 more
  1. 2022High Fidelity Neural Audio Compressioncited by 687
  2. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  3. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  4. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  5. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  6. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  7. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  8. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  9. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  10. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  11. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  12. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  13. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  14. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 218
  15. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  16. 2023Denoising Diffusion Bridge Modelscited by 178
  17. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  18. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  19. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  20. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 112
  21. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 102
  22. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  23. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 70
  24. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  25. 2024Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesiscited by 48
  26. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  27. 2024VidTok: A Versatile and Open-Source Video Tokenizercited by 35
  28. 2024GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinementcited by 34
  29. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 13
  30. 2024Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decodingcited by 10
  31. 2024Improving Vector-Quantized Image Modeling with Latent Consistency-Matching Diffusioncited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Continuous Diffusion Model for Language Modeling
  2. 2022Continuous diffusion for categorical data
  3. 2025Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models
  4. 2026Token Time Continuous Diffusion for Language Modeling
  5. 2024Unified Discrete Diffusion for Categorical Data
  6. 2026Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
  7. 2026Drifting Objectives for Refining Discrete Diffusion Language Models
  8. 2026Rethinking Token Prediction: Tree-Structured Diffusion Language Model