# DiSA: Diffusion Step Annealing in Autoregressive Image Generation arXiv: https://arxiv.org/abs/2505.20297 · HTML: https://mlsanity.com/paper/2505.20297/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. - Builds on 33 of its 39 references that are in the map. - First posted on arXiv 26 May 2025. ## Placed in - Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/) ## Builds on (most cited first, 30 of 33) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020) - [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021) - [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021) - [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022) - [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019) - [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021) - [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022) - [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020) - [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016) - [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022) - [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019) - [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024) - [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023) - [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022) - [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024) - [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022) - [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022) - [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024) - [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://mlsanity.com/paper/2305.08891/) (2023) - [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](https://mlsanity.com/paper/2412.04431/) (2024) - [FlowAR: Scale-wise Autoregressive Image Generation Meets Flow Matching](https://mlsanity.com/paper/2412.15205/) (2024) - [Harmonizing Visual Representations for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.21979/) (2025) - [Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2502.20388/) (2025) - [Continuous Speculative Decoding for Autoregressive Image Generation](https://mlsanity.com/paper/2411.11925/) (2024) - [DepthMaster: Taming Diffusion Models for Monocular Depth Estimation](https://mlsanity.com/paper/2501.02576/) (2025) - [LazyMAR: Accelerating Masked Autoregressive Models via Feature Caching](https://mlsanity.com/paper/2503.12450/) (2025) - [Generative Edge Detection with Stable Diffusion](https://mlsanity.com/paper/2410.03080/) (2024) ## Built on by (most cited first, 5 of 5) - [UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation](https://mlsanity.com/paper/2510.10575/) (2025) - [Generation then Reconstruction: Accelerating Masked Autoregressive Models via Two-Stage Sampling](https://mlsanity.com/paper/2510.17171/) (2025) - [Optimizing Input of Denoising Score Matching is Biased Towards Higher Score Norm](https://mlsanity.com/paper/2511.11727/) (2025) - [LINA: Linear Autoregressive Image Generative Models with Continuous Tokens](https://mlsanity.com/paper/2601.22630/) (2026) - [XYZFlow:Scaling Multi dimensional Shortcut Flows for Efficient Generative Modeling](https://mlsanity.com/paper/2608.12276/) (2026) ## Abstract (arXiv) An increasing number of autoregressive models, such as MAR, FlowAR, xAR, and Harmon adopt diffusion sampling to improve the quality of image generation. However, this strategy leads to low inference efficiency, because it usually takes 50 to 100 steps for diffusion to sample a token. This paper explores how to effectively address this issue. Our key motivation is that as more tokens are generated during the autoregressive process, subsequent tokens follow more constrained distributions and are easier to sample. To intuitively explain, if a model has generated part of a dog, the remaining tokens must complete the dog and thus are more constrained. Empirical evidence supports our motivation: at later generation stages, the next tokens can be well predicted by a multilayer perceptron, exhibit low variance, and follow closer-to-straight-line denoising paths from noise to tokens. Based on our finding, we introduce diffusion step annealing (DiSA), a training-free method which gradually uses fewer diffusion steps as more tokens are generated, e.g., using 50 steps at the beginning and gradually decreasing to 5 steps at later stages. Because DiSA is derived from our finding specific to diffusion in autoregressive models, it is complementary to existing acceleration methods designed for diffusion alone. DiSA can be implemented in only a few lines of code on existing models, and albeit simple, achieves 5-10× faster inference for MAR and Harmon and 1.4-2.5× for FlowAR and xAR, while maintaining the generation quality.