# RandAR: Decoder-only Autoregressive Visual Generation in Random Orders

arXiv: https://arxiv.org/abs/2412.01827 · HTML: https://mlsanity.com/paper/2412.01827/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Placed in Fast autoregressive visual generation, one of 12 directions in Efficient LLM inference and serving.
- Builds on 58 of its 67 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Placed in

- Efficient LLM inference and serving / [Fast autoregressive visual generation](https://mlsanity.com/inference/fast-autoregressive-visual-generation/)

## Builds on (most cited first, 30 of 58)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)

## Built on by (most cited first, 30 of 83)

- [Show-o2: Improved Native Unified Multimodal Models](https://mlsanity.com/paper/2506.15564/) (2025)
- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [Parallelized Autoregressive Visual Generation](https://mlsanity.com/paper/2412.15119/) (2024)
- [Unified Autoregressive Visual Generation and Understanding with Continuous Tokens](https://mlsanity.com/paper/2503.13436/) (2025)
- [Transition Models: Rethinking the Generative Learning Objective](https://mlsanity.com/paper/2509.04394/) (2025)
- [Neighboring Autoregressive Modeling for Efficient Visual Generation](https://mlsanity.com/paper/2503.10696/) (2025)
- [Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation](https://mlsanity.com/paper/2503.16430/) (2025)
- [Frequency Autoregressive Image Generation with Continuous Tokens](https://mlsanity.com/paper/2503.05305/) (2025)
- [Next Patch Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2412.15321/) (2024)
- [Next Block Prediction: Video Generation via Semi-Autoregressive Modeling](https://mlsanity.com/paper/2502.07737/) (2025)
- [Next-Embedding Prediction Makes Strong Vision Learners](https://mlsanity.com/paper/2512.16922/) (2025)
- [Visual Autoregressive Modeling for Instruction-Guided Image Editing](https://mlsanity.com/paper/2508.15772/) (2025)
- [Macro-from-Micro Planning for High-Quality and Parallelized Autoregressive Long Video Generation](https://mlsanity.com/paper/2508.03334/) (2025)
- [Autoregressive Image Generation with Randomized Parallel Decoding](https://mlsanity.com/paper/2503.10568/) (2025)
- [Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model](https://mlsanity.com/paper/2506.05289/) (2025)
- [Latent Denoising Makes Good Tokenizers](https://mlsanity.com/paper/2507.15856/) (2025)
- [Multi-Token Prediction Needs Registers](https://mlsanity.com/paper/2505.10518/) (2025)
- [Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture](https://mlsanity.com/paper/2506.19935/) (2025)
- [Hyperspherical Latents Improve Continuous-Token Autoregressive Generation](https://mlsanity.com/paper/2509.24335/) (2025)
- [SoFlow: Solution Flow Models for One-Step Generative Modeling](https://mlsanity.com/paper/2512.15657/) (2025)
- [MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization](https://mlsanity.com/paper/2504.00999/) (2025)
- [Image Tokenizer Needs Post-Training](https://mlsanity.com/paper/2509.12474/) (2025)
- [Self-Speculative Masked Diffusions](https://mlsanity.com/paper/2510.03929/) (2025)
- [Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesis](https://mlsanity.com/paper/2503.08354/) (2025)
- [Unleashing the Power of Chain-of-Prediction for Monocular 3D Object Detection](https://mlsanity.com/paper/2505.04594/) (2025)
- [Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation](https://mlsanity.com/paper/2507.01957/) (2025)
- [Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization](https://mlsanity.com/paper/2509.10140/) (2025)
- [D-AR: Diffusion via Autoregressive Models](https://mlsanity.com/paper/2505.23660/) (2025)
- [Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling](https://mlsanity.com/paper/2503.16278/) (2025)
- [Autoregressive Images Watermarking through Lexical Biasing: An Approach Resistant to Regeneration Attack](https://mlsanity.com/paper/2506.01011/) (2025)

## Abstract (arXiv)

We introduce RandAR, a decoder-only visual autoregressive (AR) model capable of generating images in arbitrary token orders. Unlike previous decoder-only AR models that rely on a predefined generation order, RandAR removes this inductive bias, unlocking new capabilities in decoder-only generation. Our essential design enables random order by inserting a "position instruction token" before each image token to be predicted, representing the spatial location of the next image token. Trained on randomly permuted token sequences -- a more challenging task than fixed-order generation, RandAR achieves comparable performance to its conventional raster-order counterpart. More importantly, decoder-only transformers trained from random orders acquire new capabilities. For the efficiency bottleneck of AR models, RandAR adopts parallel decoding with KV-Cache at inference time, enjoying 2.5x acceleration without sacrificing generation quality. Additionally, RandAR supports inpainting, outpainting and resolution extrapolation in a zero-shot manner. We hope RandAR inspires new directions for decoder-only visual generation models and broadens their applications across diverse scenarios. Our project page is at https://rand-ar.github.io/.