# Learning-Order Autoregressive Models with Application to Molecular Graph Generation

arXiv: https://arxiv.org/abs/2503.05979 · HTML: https://mlsanity.com/paper/2503.05979/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [DiGress: Discrete Denoising diffusion for graph generation](https://mlsanity.com/paper/2209.14734/) (2022)
- [GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models](https://mlsanity.com/paper/1802.08773/) (2018)
- [Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data](https://mlsanity.com/paper/2406.03736/) (2024)
- [Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations](https://mlsanity.com/paper/2202.02514/) (2022)
- [Learning Deep Generative Models of Graphs](https://mlsanity.com/paper/1803.03324/) (2018)
- [Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement](https://mlsanity.com/paper/1903.06059/) (2019)
- [Autoregressive Diffusion Models](https://mlsanity.com/paper/2110.02037/) (2021)
- [Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discovery](https://mlsanity.com/paper/1803.09518/) (2018)
- [Autoregressive Diffusion Model for Graph Generation](https://mlsanity.com/paper/2307.08849/) (2023)
- [Variational Flow Matching for Graph Generation](https://mlsanity.com/paper/2406.04843/) (2024)
- [A Deep and Tractable Density Estimator](https://mlsanity.com/paper/1310.1757/) (2013)
- [Order Matters: Probabilistic Modeling of Node Sequence for Graph Generation](https://mlsanity.com/paper/2106.06189/) (2021)
- [Gradient Estimation with Discrete Stein Operators](https://mlsanity.com/paper/2202.09497/) (2022)
- [Let There Be Order: Rethinking Ordering in Autoregressive Graph Generation](https://mlsanity.com/paper/2305.15562/) (2023)

## Built on by (most cited first, 29 of 29)

- [Discrete Diffusion in Large Language and Multimodal Models: A Survey](https://mlsanity.com/paper/2506.13759/) (2025)
- [Any-Order Flexible Length Masked Diffusion](https://mlsanity.com/paper/2509.01025/) (2025)
- [Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion](https://mlsanity.com/paper/2506.08316/) (2025)
- [Planner Aware Path Learning in Diffusion Language Models Training](https://mlsanity.com/paper/2509.23405/) (2025)
- [Rainbow Padding: Mitigating Early Termination in Instruction-Tuned Diffusion LLMs](https://mlsanity.com/paper/2510.03680/) (2025)
- [DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking](https://mlsanity.com/paper/2603.01367/) (2026)
- [Symmetry-Aware GFlowNets](https://mlsanity.com/paper/2506.02685/) (2025)
- [Unifying Masked Diffusion Models with Various Generation Orders and Beyond](https://mlsanity.com/paper/2602.02112/) (2026)
- [Masked Diffusion Models are Secretly Learned-Order Autoregressive Models](https://mlsanity.com/paper/2511.19152/) (2025)
- [NEAT: Neighborhood-Guided, Efficient, Autoregressive Set Transformer for 3D Molecular Generation](https://mlsanity.com/paper/2512.05844/) (2025)
- [DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Discrete Diffusion Models](https://mlsanity.com/paper/2604.24357/) (2026)
- [Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster](https://mlsanity.com/paper/2605.18204/) (2026)
- [TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models](https://mlsanity.com/paper/2605.24292/) (2026)
- [Variational Learning for Insertion-based Generation](https://mlsanity.com/paper/2606.02133/) (2026)
- [Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding](https://mlsanity.com/paper/2607.01775/) (2026)
- [Distilling Specialized Orders for Visual Generation](https://mlsanity.com/paper/2504.17069/) (2025)
- [Designing ReLU Generative Networks to Enumerate Trees with a Given Tree Edit Distance](https://mlsanity.com/paper/2510.10706/) (2025)
- [Neural Proposals, Symbolic Guarantees: Neuro-Symbolic Graph Generative Modeling](https://mlsanity.com/paper/2602.16954/) (2026)
- [Insertion Based Sequence Generation with Learnable Order Dynamics](https://mlsanity.com/paper/2602.18695/) (2026)
- [Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models](https://mlsanity.com/paper/2606.23567/) (2026)
- [A document is worth a structured record: Principled inductive bias design for document recognition](https://mlsanity.com/paper/2507.08458/) (2025)
- [Diffusion Models at the Drug Discovery Frontier: A Review on Generating Small Molecules versus Therapeutic Peptides](https://mlsanity.com/paper/2511.00209/) (2025)
- [Learning Generation Orders for Masked Discrete Diffusion Models via Variational Inference](https://mlsanity.com/paper/2602.23968/) (2026)
- [SIGMA: Semantic Identifier Grouping for Molecular Autoregression](https://mlsanity.com/paper/2603.25062/) (2026)
- [ReLU Networks for Exact Generation of Similar Graphs](https://mlsanity.com/paper/2604.05929/) (2026)
- [Order-Agnostic Autoregressive Modelling with Missing Data](https://mlsanity.com/paper/2605.06355/) (2026)
- [Adaptive Order Policies for Masked Diffusion](https://mlsanity.com/paper/2606.00295/) (2026)
- [Scaling Novel Graph Generation via Lightweight Structure-Guided Autoregressive Models](https://mlsanity.com/paper/2606.04287/) (2026)
- [Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design](https://mlsanity.com/paper/2606.15327/) (2026)

## Abstract (arXiv)

Autoregressive models (ARMs) have become the workhorse for sequence generation tasks, since many problems can be modeled as next-token prediction. While there appears to be a natural ordering for text (i.e., left-to-right), for many data types, such as graphs, the canonical ordering is less obvious. To address this problem, we introduce a variant of ARM that generates high-dimensional data using a probabilistic ordering that is sequentially inferred from data. This model incorporates a trainable probability distribution, referred to as an order-policy, that dynamically decides the autoregressive order in a state-dependent manner. To train the model, we introduce a variational lower bound on the log-likelihood, which we optimize with stochastic gradient estimation. We demonstrate experimentally that our method can learn meaningful autoregressive orderings in image and graph generation. On the challenging domain of molecular graph generation, we achieve state-of-the-art results on the QM9 and ZINC250k benchmarks, evaluated across key metrics for distribution similarity and drug-likeless.