mlsanityData to 14 Oct 2026

Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture

arXiv 2506.19935 ↗First posted 24 Jun 2025cs.LG cs.CV stat.MLcited by 16

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Efficiently scaling Large Language Models (LLMs) necessitates exploring alternatives to dominant autoregressive (AR) methods, with Masked Diffusion Models (MDMs) emerging as candidates. However, comparing AR (typically decoder-only) and MDM (often encoder-only) paradigms is confounded by differing architectures, obscuring true algorithmic and efficiency trade-offs. This research decouples these factors by evaluating MDMs within a decoder-only framework to: (1) Equitably compare MDM (as Any-Order AR) and standard AR paradigms through discrepancies on orders. (2) Investigate MDM architectural impacts on computational efficiency. We show decoder-only MDMs, despite a larger modeling space, can achieve significant inference speedups (~25×) and comparable perplexity with techniques like temperature annealing, offering a path to reduced inference compute. This work provides insights for developing more computationally efficient foundation models by disentangling core modeling choices from architectural influences. Code is available at https://github.com/scxue/AO-GPT-MDM.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 35 of its 48 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

35 of 48 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  2. 2025Large Language Diffusion Models14 Feb
  3. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
  4. 2024Energy-Based Diffusion Language Models for Text Generation28 Oct
  5. 2024Discrete Copula Diffusion2 Oct
  6. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  7. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  8. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  9. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  10. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  11. 2021Autoregressive Diffusion Models5 Oct

Other references · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  12. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
12 more
  1. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  2. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  3. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  4. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 307
  5. 2024Randomized Autoregressive Visual Generationcited by 127
  6. 2016Neural Autoregressive Distribution Estimationcited by 95
  7. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 83
  8. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  9. 2019Insertion-based Decoding with automatically Inferred Generation Ordercited by 43
  10. 2021Arbitrary Conditional Distributions with Energycited by 22
  11. 2024σ-GPTs: A New Approach to Autoregressive Modelscited by 20
  12. 2024ENTP: Encoder-only Next Token Predictioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025Diffusion Language Models are Super Data Learnerscited by 48
  2. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
  3. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
  4. 2025Masks Can Be Distracting: On Context Comprehension in Diffusion Language Modelscited by 9
  5. 2026Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignmentcited by 6
  6. 2025DiffGRM: Diffusion-based Generative Recommendation Modelcited by 4
  7. 2025Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMscited by 2
  8. 2026GeoBlock: Inferring Block Granularity from Dependency Geometry in Diffusion Language Modelscited by 1
  9. 2026Speculative Correction: Draft-then-Refine Decoding for Diffusion Language Modelscited by 1
  10. 2025Spanning Tree Autoregressive Visual Generationcited by 0
  11. 2025Dual-objective Language Models: Training Efficiency Without Overfittingcited by 0
  12. 2026Why Any-Order Autoregressive Models Need Two-Stream Attention: A Structural-Semantic Tradeoffcited by 0
4 more
  1. 2026Understanding and Accelerating the Training of Masked Diffusion Language Modelscited by 0
  2. 2026Simplifying the Modeling of Arbitrary Conditionals in Natural Languagecited by 0
  3. 2026Line-Coupled Language Modelcited by 0
  4. 2026STAMP as a topology-aware molecular language for fragment-level learning and editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  2. 2026Auto-Regressive Masked Diffusion Models
  3. 2026Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models
  4. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  5. 2026Scaling Beyond Masked Diffusion Language Models
  6. 2025Masked Diffusion Models are Secretly Learned-Order Autoregressive Models
  7. 2025How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices
  8. 2026Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison