mlsanityData to 14 Oct 2026

Discrete Diffusion for Generative Modeling of Text-Aligned Speech Tokens

arXiv 2509.20060 ↗First posted 24 Sep 2025eess.AScited by 7

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

This paper introduces a discrete diffusion model (DDM) framework for text-aligned speech tokenization and reconstruction. By replacing the auto-regressive speech decoder with a discrete diffusion counterpart, our model achieves significantly better reconstruction quality, stronger ASR performance, and faster inference. We provide a comprehensive analysis of applying DDMs to speech reconstruction, examining sampler choices, inference steps, and robustness to length-scale estimation errors. Furthermore, we improve the original TASTE by systematically comparing vector quantization modules, showing that FSQ yields up to a 35% relative WER reduction and +0.14 UT-MOS improvement over RVQ for AR models, while also enhancing DDM performance. Our model generates speech in just 10 denoising steps and even supports single-step generation with only minor quality degradation.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 32 of its 33 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

32 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Token-Based Audio Inpainting via Discrete Diffusion11 Jul
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
  3. 2024Vector Quantized Diffusion Model Based Speech Bandwidth Extension9 Sep
  4. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  5. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul

Other references · 27

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2025Large Language Diffusion Modelscited by 912
  6. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  8. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  9. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  10. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  11. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  12. 2024Discrete Flow Matchingcited by 317
15 more
  1. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  2. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  3. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  4. 2025Remasking Discrete Diffusion Models with Inference-Time Scalingcited by 189
  5. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  6. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 131
  7. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 124
  8. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  9. 2021TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global contextcited by 62
  10. 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 59
  11. 2025Edit Flows: Flow Matching with Edit Operationscited by 55
  12. 2022HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancementcited by 49
  13. 2025Granary: Speech Recognition and Translation Dataset in 25 European Languagescited by 22
  14. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  15. 2025Training and Inference Efficiency of Encoder-Decoder Speech Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026The Diffusion Duality, Chapter II: Psi-Samplerscited by 12
  2. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  3. 2026Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTScited by 3
  4. 2026TASTE-Streaming: Towards Streamable Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 1
  5. 2026Mask, Sample, Revise: A Revisable CTMC Inference Stack for Guided Discrete Flow Matching Text-to-Speechcited by 0
  6. 2026Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editingcited by 0
  7. 2026StreamAlign: Streaming Text-Aligned Speech Tokenizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling
  2. 2026Revise-TTS: A Non-Masking Refinement Framework for Masked Diffusion Models in Text-to-Speech
  3. 2024Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study
  4. 2026DLLM-TTS: Block Discrete Diffusion Language Model for Text-to-Speech Synthesis
  5. 2025DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding
  6. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  7. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis
  8. 2022ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech