mlsanityData to 14 Oct 2026

DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation

arXiv 2502.03930 ↗First posted 6 Feb 2025eess.AS cs.AI cs.CLcited by 57

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Several recent studies have attempted to autoregressively generate continuous speech representations without discrete speech tokens by combining diffusion and autoregressive models, yet they often face challenges with excessive computational loads or suboptimal outcomes. In this work, we propose Diffusion Transformer Autoregressive Modeling (DiTAR), a patch-based autoregressive framework combining a language model with a diffusion transformer. This approach significantly enhances the efficacy of autoregressive models for continuous tokens and reduces computational demands. DiTAR utilizes a divide-and-conquer strategy for patch generation, where the language model processes aggregated patch embeddings and the diffusion transformer subsequently generates the next patch based on the output of the language model. For inference, we propose defining temperature as the time point of introducing noise during the reverse diffusion ODE to balance diversity and determinism. We also show in the extensive scaling analysis that DiTAR has superb scalability. In zero-shot speech generation, DiTAR achieves state-of-the-art performance in robustness, speaker similarity, and naturalness.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 56 of its 62 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

56 of 62 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  2. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis8 Jun
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  4. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speech2 Nov

Other references · 52

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2022Flow Matching for Generative Modelingcited by 5,702
  11. 2024Qwen2.5 Technical Reportcited by 4,942
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
38 more
  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019Root Mean Square Layer Normalizationcited by 1,480
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  9. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  10. 2020On Layer Normalization in the Transformer Architecturecited by 922
  11. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  12. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  13. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  14. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  15. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  16. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  17. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  18. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  19. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  20. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  21. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  22. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  23. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  24. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  25. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 283
  26. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  27. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  28. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  29. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  30. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  31. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  32. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 118
  33. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 94
  34. 2023Stay on topic with Classifier-Free Guidancecited by 91
  35. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  36. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  37. 2023LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Modelscited by 49
  38. 2023Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesiscited by 41

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 38.

57 papers.

Sort:

By area:

  1. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  2. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  3. 2025VibeVoice Technical Reportcited by 29
  4. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 25
  5. 2025SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinementcited by 23
  6. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 22
  7. 2025Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representationcited by 17
  8. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 16
  9. 2025DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matchingcited by 15
  10. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  11. 2025Step-Audio-EditX Technical Reportcited by 7
  12. 2026IndexTTS 2.5 Technical Reportcited by 6
38 more
  1. 2025Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesiscited by 5
  2. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 5
  3. 2026ReStyle-TTS: Relative and Continuous Style Control for Zero-Shot Speech Synthesiscited by 5
  4. 2025Direct Preference Optimization for Speech Autoregressive Diffusion Modelscited by 4
  5. 2026dots.tts Technical Reportcited by 4
  6. 2025StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modelingcited by 3
  7. 2025Next Tokens Denoising for Speech Synthesiscited by 3
  8. 2025DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generationcited by 3
  9. 2026Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesiscited by 3
  10. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  11. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Spacecited by 3
  12. 2026AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Followingcited by 3
  13. 2025Towards Flow-Matching-based TTS without Classifier-Free Guidancecited by 2
  14. 2025MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesiscited by 2
  15. 2026Sylber 2.0: A Universal Syllable Embeddingcited by 2
  16. 2026HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understandingcited by 2
  17. 2026FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generationcited by 2
  18. 2026ARCHI-TTS: A flow-matching-based Text-to-Speech Model with Self-supervised Semantic Aligner and Accelerated Inferencecited by 1
  19. 2026Meltdown: Circuits and Bifurcations in Point-Cloud-Conditioned 3D Diffusion Transformerscited by 1
  20. 2026WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modelingcited by 1
  21. 2026F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generationcited by 1
  22. 2026DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speechcited by 1
  23. 2026GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech modelcited by 1
  24. 2026AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editingcited by 1
  25. 2025DiFlow-TTS: Compact and Low-Latency Zero-Shot Text-to-Speech with Discrete Flow Matchingcited by 0
  26. 2025RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speechcited by 0
  27. 2026Decoding Order Matters in Autoregressive Speech Synthesiscited by 0
  28. 2026RobustSpeechFlow: Learning Robust Text-to-Speech Trajectories via Augmentation-based Contrastive Flow Matchingcited by 0
  29. 2026PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesiscited by 0
  30. 2026TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speechcited by 0
  31. 2026HoliDubber: Holistic Video Dubbing for Complex Acoustic Scenes via Text-Guided Audio Synthesiscited by 0
  32. 2026FreyaTTS: A Compact Tokenizer-Free Flow-Matching Transformer for Turkish-First Speech Synthesiscited by 0
  33. 2026VocalRender: Score-Native Singing Voice Synthesis for Real-World Compositioncited by 0
  34. 2026Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokenscited by 0
  35. 2026DLLM-TTS: Block Discrete Diffusion Language Model for Text-to-Speech Synthesiscited by 0
  36. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latentscited by 0
  37. 2026Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenizationcited by 0
  38. 2026VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matchingcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation
  2. 2023DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation
  3. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
  4. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
  5. 2025Discrete Diffusion Models for Language Generation
  6. 2024DPI-TTS: Directional Patch Interaction for Fast-Converging and Style Temporal Modeling in Text-to-Speech
  7. 2024Improving Robustness of Diffusion-Based Zero-Shot Speech Synthesis via Stable Formant Generation
  8. 2024Speech Synthesis From Continuous Features Using Per-Token Latent Diffusion