# Randomized Autoregressive Visual Generation

arXiv: https://arxiv.org/abs/2411.00776 · HTML: https://mlsanity.com/paper/2411.00776/

## Key facts

- Cited by 127 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 81 references that are in the map.
- First posted on arXiv 1 Nov 2024.

## Builds on (most cited first, 30 of 65)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)

## Built on by (most cited first, 30 of 127)

- [Improved Mean Flows: On the Challenges of Fastforward Generative Models](https://mlsanity.com/paper/2512.02012/) (2025)
- [DDT: Decoupled Diffusion Transformer](https://mlsanity.com/paper/2504.05741/) (2025)
- [Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation](https://mlsanity.com/paper/2505.05472/) (2025)
- [MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft](https://mlsanity.com/paper/2504.08388/) (2025)
- [Masked Autoencoders Are Effective Tokenizers for Diffusion Models](https://mlsanity.com/paper/2502.03444/) (2025)
- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [RandAR: Decoder-only Autoregressive Visual Generation in Random Orders](https://mlsanity.com/paper/2412.01827/) (2024)
- [FlowAR: Scale-wise Autoregressive Image Generation Meets Flow Matching](https://mlsanity.com/paper/2412.15205/) (2024)
- [Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2502.20388/) (2025)
- [A Bias-Free Training Paradigm for More General AI-generated Image Detection](https://mlsanity.com/paper/2412.17671/) (2024)
- [Diffusion Beats Autoregressive in Data-Constrained Settings](https://mlsanity.com/paper/2507.15857/) (2025)
- [SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer](https://mlsanity.com/paper/2412.10958/) (2024)
- [GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning](https://mlsanity.com/paper/2505.17022/) (2025)
- [Unified Autoregressive Visual Generation and Understanding with Continuous Tokens](https://mlsanity.com/paper/2503.13436/) (2025)
- [Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens](https://mlsanity.com/paper/2501.07730/) (2025)
- [Puppeteer: Rig and Animate Your 3D Models](https://mlsanity.com/paper/2508.10898/) (2025)
- [Diffusion Models without Classifier-free Guidance](https://mlsanity.com/paper/2502.12154/) (2025)
- [FlowTok: Flowing Seamlessly Across Text and Image Tokens](https://mlsanity.com/paper/2503.10772/) (2025)
- [Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning](https://mlsanity.com/paper/2505.07538/) (2025)
- [Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey](https://mlsanity.com/paper/2412.18619/) (2024)
- [Frequency Autoregressive Image Generation with Continuous Tokens](https://mlsanity.com/paper/2503.05305/) (2025)
- ["Principal Components" Enable A New Language of Images](https://mlsanity.com/paper/2503.08685/) (2025)
- [Next Patch Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2412.15321/) (2024)
- [Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator](https://mlsanity.com/paper/2503.01103/) (2025)
- [Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspective](https://mlsanity.com/paper/2507.08801/) (2025)
- [Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2507.08441/) (2025)
- [LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimization](https://mlsanity.com/paper/2503.08619/) (2025)
- [Autoregressive Image Generation with Randomized Parallel Decoding](https://mlsanity.com/paper/2503.10568/) (2025)
- [Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model](https://mlsanity.com/paper/2506.05289/) (2025)
- [Latent Denoising Makes Good Tokenizers](https://mlsanity.com/paper/2507.15856/) (2025)

## Abstract (arXiv)

This paper presents Randomized AutoRegressive modeling (RAR) for visual generation, which sets a new state-of-the-art performance on the image generation task while maintaining full compatibility with language modeling frameworks. The proposed RAR is simple: during a standard autoregressive training process with a next-token prediction objective, the input sequence-typically ordered in raster form-is randomly permuted into different factorization orders with a probability r, where r starts at 1 and linearly decays to 0 over the course of training. This annealing training strategy enables the model to learn to maximize the expected likelihood over all factorization orders and thus effectively improve the model's capability of modeling bidirectional contexts. Importantly, RAR preserves the integrity of the autoregressive modeling framework, ensuring full compatibility with language modeling while significantly improving performance in image generation. On the ImageNet-256 benchmark, RAR achieves an FID score of 1.48, not only surpassing prior state-of-the-art autoregressive image generators but also outperforming leading diffusion-based and masked transformer-based methods. Code and models will be made available at https://github.com/bytedance/1d-tokenizer