mlsanityData to 14 Oct 2026

Next Tokens Denoising for Speech Synthesis

arXiv 2507.22746 ↗First posted 30 Jul 2025cs.SD cs.CL eess.AScited by 3

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

While diffusion and autoregressive (AR) models have significantly advanced generative modeling, they each present distinct limitations. AR models, which rely on causal attention, cannot exploit future context and suffer from slow generation speeds. Conversely, diffusion models struggle with key-value (KV) caching. To overcome these challenges, we introduce Dragon-FM, a novel text-to-speech (TTS) design that unifies AR and flow-matching. This model processes 48 kHz audio codec tokens in chunks at a compact rate of 12.5 tokens per second. This design enables AR modeling across chunks, ensuring global coherence, while parallel flow-matching within chunks facilitates fast iterative denoising. Thus, the model leverages KV-cache across chunks and utilizes bidirectional context within each chunk. Furthermore, it bridges continuous and discrete feature modeling, demonstrating that continuous AR flow-matching can predict discrete tokens with finite scalar quantizers. This efficient codec and fast chunk-autoregressive architecture also make the model highly effective for generating long-form content, such as podcasts. Experiments on podcast datasets demonstrate its capability to efficiently generate high-quality zero-shot podcasts.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 52 of its 57 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

52 of 57 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching16 Feb
  2. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation6 Feb
  3. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis8 Jun
  4. 2024Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study7 Jun
  5. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  6. 2023Better speech synthesis through scaling12 May
  7. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  8. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusion27 Jan

Other references · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 879
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  10. 2022High Fidelity Neural Audio Compressioncited by 687
  11. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  12. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
32 more
  1. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  2. 2025Mean Flows for One-step Generative Modelingcited by 561
  3. 2023Simple and Controllable Music Generationcited by 488
  4. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  5. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 395
  6. 2024One Step Diffusion via Shortcut Modelscited by 391
  7. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  8. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  9. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  10. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  11. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  12. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  13. 2024PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Acceleratorcited by 97
  14. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  15. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  16. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 48
  17. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
  18. 2022Understanding DDPM Latent Codes Through Optimal Transportcited by 45
  19. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  20. 2022iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transformcited by 32
  21. 2025Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspectivecited by 22
  22. 2022DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoderscited by 15
  23. 2018Neural Speech Synthesis with Transformer Networkcited by 14
  24. 2022Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speechcited by 13
  25. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  26. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  27. 2023FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Modelcited by 7
  28. 2024Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learnerscited by 5
  29. 2024Investigating Neural Audio Codecs for Speech Language Model-Based Speech Generationcited by 4
  30. 2020MoBoAligner: a Neural Alignment Model for Non-autoregressive TTS with Monotonic Boundary Searchcited by 3
  31. 2025Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modelingcited by 3
  32. 2025StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modelingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Direct Preference Optimization for Speech Autoregressive Diffusion Modelscited by 4
  2. 2026Sylber 2.0: A Universal Syllable Embeddingcited by 2
  3. 2026Investigating Group Relative Policy Optimization for Diffusion Transformer based Text-to-Audio Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Autoregressive Audio Modeling via Next-Scale Prediction
  2. 2026Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation ↗
  3. 2026TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech
  4. 2024LAFMA: A Latent Flow Matching Model for Text-to-Audio Generation
  5. 2026DLLM-TTS: Block Discrete Diffusion Language Model for Text-to-Speech Synthesis
  6. 2025Discrete Diffusion for Generative Modeling of Text-Aligned Speech Tokens
  7. 2025DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation
  8. 2025CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis