# Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer

arXiv: https://arxiv.org/abs/2206.04452 · HTML: https://mlsanity.com/paper/2206.04452/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis](https://mlsanity.com/paper/2108.08827/) (2021)
- [Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2008.07905/) (2020)
- [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021)

## Built on by (most cited first, 30 of 30)

- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [An Image is Worth 32 Tokens for Reconstruction and Generation](https://mlsanity.com/paper/2406.07550/) (2024)
- [MaskBit: Embedding-free Image Generation via Bit Tokens](https://mlsanity.com/paper/2409.16211/) (2024)
- [Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization](https://mlsanity.com/paper/2305.11718/) (2023)
- [Temporally Consistent Transformers for Video Generation](https://mlsanity.com/paper/2210.02396/) (2022)
- [HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes](https://mlsanity.com/paper/2401.00365/) (2023)
- [StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis](https://mlsanity.com/paper/2401.17093/) (2024)
- [Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis](https://mlsanity.com/paper/2406.05478/) (2024)
- [Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective](https://mlsanity.com/paper/2410.12490/) (2024)
- [σ-GPTs: A New Approach to Autoregressive Models](https://mlsanity.com/paper/2404.09562/) (2024)
- [MAGVLT: Masked Generative Vision-and-Language Transformer](https://mlsanity.com/paper/2303.12208/) (2023)
- [AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation](https://mlsanity.com/paper/2409.00342/) (2024)
- [StraIT: Non-autoregressive Generation with Stratified Image Transformer](https://mlsanity.com/paper/2303.00750/) (2023)
- [ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis](https://mlsanity.com/paper/2411.06959/) (2024)
- [Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers](https://mlsanity.com/paper/2303.11251/) (2023)
- [Masked Generative Priors Improve World Models Sequence Modelling Capabilities](https://mlsanity.com/paper/2410.07836/) (2024)
- [Accurate and Efficient World Modeling with Masked Latent Transformers](https://mlsanity.com/paper/2507.04075/) (2025)
- [Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models](https://mlsanity.com/paper/2304.01515/) (2023)
- [Plug-and-Play Context Feature Reuse for Efficient Masked Generation](https://mlsanity.com/paper/2505.19089/) (2025)
- [Progressive Text-to-Image Generation](https://mlsanity.com/paper/2210.02291/) (2022)
- [Aerial Image Object Detection With Vision Transformer Detector (ViTDet)](https://mlsanity.com/paper/2301.12058/) (2023)
- [Layton: Latent Consistency Tokenizer for 1024-pixel Image Reconstruction and Generation by 256 Tokens](https://mlsanity.com/paper/2503.08377/) (2025)
- [Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering](https://mlsanity.com/paper/2606.01911/) (2026)
- [Emage: Non-Autoregressive Text-to-Image Generation](https://mlsanity.com/paper/2312.14988/) (2023)
- [AdaGen: Learning Adaptive Policy for Image Synthesis](https://mlsanity.com/paper/2603.06993/) (2026)
- [Variational Distribution Learning for Unsupervised Text-to-Image Generation](https://mlsanity.com/paper/2303.16105/) (2023)
- [Towards Multi-Task Multi-Modal Models: A Video Generative Perspective](https://mlsanity.com/paper/2405.16728/) (2024)
- [Improving Detail in Pluralistic Image Inpainting with Feature Dequantization](https://mlsanity.com/paper/2412.01046/) (2024)
- [Logit Refiner: Improving Visual Autoregressive Models via Intra-Scale Dependency Modeling](https://mlsanity.com/paper/2609.11804/) (2026)

## Abstract (arXiv)

Although autoregressive models have achieved promising results on image generation, their unidirectional generation process prevents the resultant images from fully reflecting global contexts. To address the issue, we propose an effective image generation framework of Draft-and-Revise with Contextual RQ-transformer to consider global contexts during the generation process. As a generalized VQ-VAE, RQ-VAE first represents a high-resolution image as a sequence of discrete code stacks. After code stacks in the sequence are randomly masked, Contextual RQ-Transformer is trained to infill the masked code stacks based on the unmasked contexts of the image. Then, Contextual RQ-Transformer uses our two-phase decoding, Draft-and-Revise, and generates an image, while exploiting the global contexts of the image during the generation process. Specifically. in the draft phase, our model first focuses on generating diverse images despite rather low quality. Then, in the revise phase, the model iteratively improves the quality of images, while preserving the global contexts of generated images. In experiments, our method achieves state-of-the-art results on conditional image generation. We also validate that the Draft-and-Revise decoding can achieve high performance by effectively controlling the quality-diversity trade-off in image generation.