mlsanityData to 7 Oct 2026

AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation

arXiv 2305.09515 ↗First posted 16 May 2023cs.CLcited by 113

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing language models are trained with a left-to-right auto-regressive approach. To account for the inherent sequential characteristic of natural language, we introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right. In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-Diffusion clearly demonstrated its superiority over existing diffusion language models and that it can be 100×~600× faster when achieving comparable results. Our code is available at https://github.com/microsoft/ProphetNet/tree/master/AR-diffusion.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 39 of its 46 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

39 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  2. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise22 Dec
  3. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers20 Dec
  4. 2022Continuous diffusion for categorical data28 Nov
  5. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct
  6. 2021Autoregressive Diffusion Models5 Oct

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
21 more
  1. 2018Hierarchical Neural Story Generationcited by 1,182
  2. 2015Teaching Machines to Read and Comprehendcited by 1,039
  3. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  4. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  5. 2017Non-Autoregressive Neural Machine Translationcited by 457
  6. 2022Understanding Diffusion Models: A Unified Perspectivecited by 337
  7. 2021Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecastingcited by 332
  8. 2015LSTM: A Search Space Odysseycited by 315
  9. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  10. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  11. 2016Incorporating Copying Mechanism in Sequence-to-Sequence Learningcited by 217
  12. 2019Levenshtein Transformercited by 197
  13. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  14. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
  15. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  16. 2020GLGE: A New General Language Generation Evaluation Benchmarkcited by 47
  17. 2020Lexically Constrained Neural Machine Translation with Levenshtein Transformercited by 45
  18. 2020BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretrainingcited by 33
  19. 2021Non-Autoregressive Translation by Learning Target Categorical Codescited by 25
  20. 2022ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generationcited by 10
  21. 2022Multi-Granularity Optimization for Non-Autoregressive Translationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 25; 2025: 49; 2026: 34.

113 papers.

Sort:

By area:

  1. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  2. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  3. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  4. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  5. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  6. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  7. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  8. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  9. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  10. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  11. 2024Rolling Diffusion Modelscited by 102
  12. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 93
38 more
  1. 2024Physics-Informed Diffusion Modelscited by 88
  2. 2025A Survey on Diffusion Language Modelscited by 75
  3. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  4. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 56
  5. 2024Discrete Copula Diffusioncited by 53
  6. 2025AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusioncited by 53
  7. 2024Streaming Diffusion Policy: Fast Policy Synthesis with Variable Noise Diffusion Modelscited by 51
  8. 2024How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Frameworkcited by 48
  9. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  10. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  11. 2023CodeFusion: A Pre-trained Diffusion Model for Code Generationcited by 31
  12. 2024On the Challenges and Opportunities in Generative AIcited by 31
  13. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 30
  14. 2026ELF: Embedded Language Flowscited by 27
  15. 2025Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modelingcited by 24
  16. 2025Diffusion vs. Autoregressive Language Models: A Text Embedding Perspectivecited by 23
  17. 2024TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodingscited by 22
  18. 2025SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinementcited by 21
  19. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inferencecited by 21
  20. 2025Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Frameworkcited by 20
  21. 2024Boosting Latent Diffusion with Perceptual Objectivescited by 19
  22. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 19
  23. 2026INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modelingcited by 18
  24. 2023Continuous Diffusion for Mixed-Type Tabular Datacited by 16
  25. 2025Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Stepcited by 16
  26. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  27. 2024On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Modelscited by 12
  28. 2025MaskFlow: Discrete Flows For Flexible and Efficient Long Video Generationcited by 12
  29. 2025RayFlow: Instance-Aware Diffusion Acceleration via Adaptive Flow Trajectoriescited by 12
  30. 2025Elucidated Rolling Diffusion Models for Probabilistic Forecasting of Complex Dynamicscited by 12
  31. 2024Efficient Autoregressive Audio Modeling via Next-Scale Predictioncited by 11
  32. 2025Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Positioncited by 11
  33. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Modelscited by 10
  34. 2025From Virtual Games to Real-World Playcited by 9
  35. 2023Generative AI in the Construction Industry: Opportunities & Challengescited by 8
  36. 2024CL-DiffPhyCon: Closed-loop Diffusion Control of Complex Physical Systemscited by 8
  37. 2024DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreakcited by 8
  38. 2025Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Modelscited by 8

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Self-conditioned Embedding Diffusion for Text Generation
  2. 2023Diffusion Models for Non-autoregressive Text Generation: A Survey
  3. 2023A Survey of Diffusion Models in Natural Language Processing
  4. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  5. 2025Discrete Diffusion Models for Language Generation
  6. 2024LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation?
  7. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
  8. 2024Energy-Based Diffusion Language Models for Text Generation