mlsanityData to 7 Oct 2026

Autoregressive Diffusion Models

arXiv 2110.02037 ↗First posted 5 Oct 2021cs.LG stat.MLcited by 208

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

We introduce Autoregressive Diffusion Models (ARDMs), a model class encompassing and generalizing order-agnostic autoregressive models (Uria et al., 2014) and absorbing discrete diffusion (Austin et al., 2021), which we show are special cases of ARDMs under mild assumptions. ARDMs are simple to implement and easy to train. Unlike standard ARMs, they do not require causal masking of model representations, and can be trained using an efficient objective similar to modern probabilistic diffusion models that scales favourably to highly-dimensional data. At test time, ARDMs support parallel generation which can be adapted to fit any given generation budget. We find that ARDMs require significantly fewer steps than discrete diffusion models to attain the same performance. Finally, we apply ARDMs to lossless compression, and show that they are uniquely suited to this task. Contrary to existing approaches based on bits-back coding, ARDMs obtain compelling results not only on complete datasets, but also on compressing single data points. Moreover, this can be done using a modest number of network calls for (de)compression due to the model's adaptable parallel generation.

Key facts. Cited by 208 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 46 of its 60 references that are in the map. First posted on arXiv 5 Oct 2021.

Builds on

46 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models16 Jul

Other references · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
33 more
  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  2. 2021Variational Diffusion Modelscited by 1,145
  3. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  4. 2016Pixel Recurrent Neural Networkscited by 814
  5. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  6. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  7. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  8. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  9. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  10. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  11. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  12. 2018Efficient Neural Audio Synthesiscited by 218
  13. 2021Gotta Go Fast When Generating Data with Score-Based Modelscited by 218
  14. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  15. 2018Constrained Graph Variational Autoencoders for Molecule Designcited by 182
  16. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matchingcited by 182
  17. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
  18. 2021Learning to Efficiently Sample from Diffusion Probabilistic Modelscited by 129
  19. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  20. 2018Practical Full Resolution Learned Lossless Image Compressioncited by 93
  21. 2019Practical Lossless Compression with Latent Variables using Bits Back Codingcited by 85
  22. 2019Integer Discrete Flows and Lossless Compressioncited by 82
  23. 2018Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscalingcited by 74
  24. 2017Parallel Multiscale Autoregressive Density Estimationcited by 71
  25. 2013A Deep and Tractable Density Estimatorcited by 67
  26. 2020Accelerating Feedforward Computation via Parallel Nonlinear Equation Solvingcited by 51
  27. 2021Consistency Regularization for Variational Auto-Encoderscited by 50
  28. 2019HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Modelscited by 36
  29. 2020IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compressioncited by 35
  30. 2019Compression with Flows via Local Bits-Back Codingcited by 34
  31. 2020Locally Masked Convolution for Autoregressive Modelscited by 22
  32. 2020Predictive Sampling with Forecasting Autoregressive Modelscited by 10
  33. 2021The DEformer: An Order-Agnostic Distribution Estimating Transformercited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 25; 2023: 34; 2024: 41; 2025: 59; 2026: 48.

208 papers.

Sort:

By area:

  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  2. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  3. 2025Large Language Diffusion Modelscited by 872
  4. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  5. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  6. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  7. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  8. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  9. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  10. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  11. 2022A Survey on Generative Diffusion Modelcited by 238
  12. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
38 more
  1. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  2. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 204
  3. 2022Continuous diffusion for categorical datacited by 202
  4. 2024Scaling up Masked Diffusion Models on Textcited by 195
  5. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  6. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 167
  7. 2022Lossy Image Compression with Conditional Diffusion Modelscited by 165
  8. 2023Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restorationcited by 164
  9. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  10. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  11. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  12. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  13. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 124
  14. 20234M: Massively Multimodal Masked Modelingcited by 118
  15. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 113
  16. 2022Latent Diffusion for Language Generationcited by 110
  17. 2024Dirichlet Flow Matching with Applications to DNA Sequence Designcited by 110
  18. 2022Soft Diffusion: Score Matching for General Corruptionscited by 101
  19. 2023Input Perturbation Reduces Exposure Bias in Diffusion Modelscited by 88
  20. 2024Energy-Based Diffusion Language Models for Text Generationcited by 86
  21. 2023Autoregressive Diffusion Model for Graph Generationcited by 82
  22. 2023Reflected Diffusion Modelscited by 81
  23. 2022An Introduction to Neural Data Compressioncited by 78
  24. 2023Dirichlet Diffusion Score Model for Biological Sequence Generationcited by 73
  25. 2021Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codescited by 71
  26. 2022Diffusion Models for Graphs Benefit From Discrete State Spacescited by 69
  27. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  28. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  29. 2023Bayesian Flow Networkscited by 62
  30. 2023AI-Generated Content (AIGC): A Surveycited by 59
  31. 2024Informed Correctors for Discrete Diffusion Modelscited by 57
  32. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Modelcited by 56
  33. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 54
  34. 2025Edit Flows: Flow Matching with Edit Operationscited by 53
  35. 2022Efficient Diffusion Models for Vision: A Surveycited by 50
  36. 2024Generative Modeling of Molecular Dynamics Trajectoriescited by 48
  37. 2024How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Frameworkcited by 48
  38. 2024Machine Unlearning for Image-to-Image Generative Modelscited by 47

50 most cited of 208 All 208 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Control-Augmented Autoregressive Diffusion for Data Assimilation
  2. 2025Diffusion Beats Autoregressive in Data-Constrained Settings
  3. 2025Capturing Conditional Dependence via Auto-regressive Diffusion Models
  4. 2025Autoregressive Distillation of Diffusion Transformers
  5. 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing
  6. 2026Optimizers for Diffusion Models: A Controlled Benchmark
  7. 2025Double Descent as a Lens for Sample Efficiency in Autoregressive vs. Discrete Diffusion Models
  8. 2026Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models