mlsanityData to 7 Oct 2026

Training and Inference on Any-Order Autoregressive Models the Right Way

arXiv 2205.13554 ↗First posted 26 May 2022cs.LG cs.AIcited by 64

Not on the map. Why?

Cited by 41 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Conditional inference on arbitrary subsets of variables is a core problem in probabilistic inference with important applications such as masked language modeling and image inpainting. In recent years, the family of Any-Order Autoregressive Models (AO-ARMs) -- closely related to popular models such as BERT and XLNet -- has shown breakthrough performance in arbitrary conditional tasks across a sweeping range of domains. But, in spite of their success, in this paper we identify significant improvements to be made to previous formulations of AO-ARMs. First, we show that AO-ARMs suffer from redundancy in their probabilistic model, i.e., they define the same distribution in multiple different ways. We alleviate this redundancy by training on a smaller set of univariate conditionals that still maintains support for efficient arbitrary conditional inference. Second, we upweight the training loss for univariate conditionals that are evaluated more frequently during inference. Our method leads to improved performance with no compromises on tractability, giving state-of-the-art likelihoods in arbitrary conditional modeling on text (Text8), image (CIFAR10, ImageNet32), and continuous tabular data domains.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

33 of 56 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  12. 2021Variational Diffusion Modelscited by 1,145
21 more
  1. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  2. 2018Image Transformercited by 770
  3. 2021Implicit Behavioral Cloningcited by 442
  4. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  5. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  6. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  7. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  8. 2021Autoregressive Diffusion Modelscited by 208
  9. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  10. 2016Semantic Image Inpainting with Deep Generative Modelscited by 156
  11. 2022Generative Flow Networks for Discrete Probabilistic Modelingcited by 113
  12. 2016Neural Autoregressive Distribution Estimationcited by 94
  13. 2013A Differential Approach to Inference in Bayesian Networkscited by 93
  14. 2013Loopy Belief Propagation for Approximate Inference: An Empirical Studycited by 76
  15. 2013A Deep and Tractable Density Estimatorcited by 67
  16. 2019Flow Models for Arbitrary Conditional Likelihoodscited by 30
  17. 2019Controlling Assistive Robots with Learned Latent Actionscited by 28
  18. 2019SPFlow: An Easy and Extensible Library for Deep Probabilistic Learning using Sum-Product Networkscited by 26
  19. 2021Arbitrary Conditional Distributions with Energycited by 22
  20. 2021Discovering Non-monotonic Autoregressive Orderings with Variational Inferencecited by 15
  21. 2019Sequence Modeling with Unconstrained Generation Ordercited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 10; 2025: 29; 2026: 20.

64 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  4. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  5. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  6. 2022Continuous diffusion for categorical datacited by 202
  7. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 148
  8. 2024Energy-Based Diffusion Language Models for Text Generationcited by 86
  9. 2025Path Planning for Masked Diffusion Model Samplingcited by 79
  10. 2023Bayesian Flow Networkscited by 62
  11. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 33
  12. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 32
38 more
  1. 2025Continuous Diffusion Model for Language Modelingcited by 29
  2. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusioncited by 25
  3. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policiescited by 21
  4. 2025CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generationcited by 17
  5. 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecturecited by 16
  6. 2025Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decodingcited by 14
  7. 2025Planner Aware Path Learning in Diffusion Language Models Trainingcited by 14
  8. 2025TabularARGN: A Flexible and Efficient Auto-Regressive Framework for Generating High-Fidelity Synthetic Datacited by 13
  9. 2024[MASK] is All You Needcited by 11
  10. 2025Diffusion Models are Secretly Exchangeable: Parallelizing DDPMs via Autospeculationcited by 11
  11. 2025Self-Speculative Masked Diffusionscited by 11
  12. 2025dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learningcited by 10
  13. 2024Plug-and-Play Controllable Generation for Discrete Masked Modelscited by 9
  14. 2025Enabling Approximate Joint Sampling in Diffusion LMscited by 9
  15. 2025Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flowscited by 8
  16. 2026Reasoning with Latent Tokens in Diffusion Language Modelscited by 8
  17. 2024Probabilistic Neural Circuitscited by 7
  18. 2024Parallel Sampling via Countingcited by 7
  19. 2025Partition Generative Modeling: Masked Modeling Without Maskscited by 6
  20. 2025Discrete Guidance Matching: Exact Guidance for Discrete Flow Matchingcited by 6
  21. 2026Unifying Masked Diffusion Models with Various Generation Orders and Beyondcited by 6
  22. 2026Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metricscited by 6
  23. 2025Non-Markovian Discrete Diffusion with Causal Language Modelscited by 5
  24. 2025Parallel Sampling via Autospeculationcited by 5
  25. 2026S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculationcited by 5
  26. 2026Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignmentcited by 5
  27. 2023Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectivescited by 4
  28. 2023Generative Marginalization Modelscited by 4
  29. 2024Unmasking Trees for Tabular Datacited by 4
  30. 2025Variational Masked Diffusion Modelscited by 4
  31. 2026Auto-Regressive Masked Diffusion Modelscited by 4
  32. 2025ShortListing Model: A Streamlined SimplexDiffusion for Discrete Variable Generationcited by 3
  33. 2025DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generationcited by 3
  34. 2025Information-Theoretic Discrete Diffusioncited by 3
  35. 2026Discrete Stochastic Localization for Non-autoregressive Generationcited by 3
  36. 2026Generalized Discrete Diffusion from Snapshotscited by 3
  37. 2026Do Tabular Foundation Models Agree with Themselves?cited by 3
  38. 2026Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decodingcited by 2

50 most cited of 64 All 64 (JSON)