# Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis

arXiv: https://arxiv.org/abs/2406.05478 · HTML: https://mlsanity.com/paper/2406.05478/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 46 of its 51 references that are in the map.
- First posted on arXiv 8 Jun 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)

## Built on by (most cited first, 22 of 22)

- [Bridging the Divide: Reconsidering Softmax and Linear Attention](https://mlsanity.com/paper/2412.06590/) (2024)
- [TinyFusion: Diffusion Transformers Learned Shallow](https://mlsanity.com/paper/2412.01199/) (2024)
- [Halton Scheduler For Masked Generative Image Transformer](https://mlsanity.com/paper/2503.17076/) (2025)
- [Real-world Image Dehazing with Coherence-based Pseudo Labeling and Cooperative Unfolding Network](https://mlsanity.com/paper/2406.07966/) (2024)
- [Customize Your Visual Autoregressive Recipe with Set Autoregressive Modeling](https://mlsanity.com/paper/2410.10511/) (2024)
- [MagicTryOn: Harnessing Diffusion Transformer for Garment-Preserving Video Virtual Try-on](https://mlsanity.com/paper/2505.21325/) (2025)
- [ELT: Elastic Looped Transformers for Visual Generation](https://mlsanity.com/paper/2604.09168/) (2026)
- [AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation](https://mlsanity.com/paper/2409.00342/) (2024)
- [Autoregressive Distillation of Diffusion Transformers](https://mlsanity.com/paper/2504.11295/) (2025)
- [MaskMamba: A Hybrid Mamba-Transformer Model for Masked Image Generation](https://mlsanity.com/paper/2409.19937/) (2024)
- [ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis](https://mlsanity.com/paper/2411.06959/) (2024)
- [CODA: Repurposing Continuous VAEs for Discrete Tokenization](https://mlsanity.com/paper/2503.17760/) (2025)
- [Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots](https://mlsanity.com/paper/2505.20288/) (2025)
- [Plug-and-Play Context Feature Reuse for Efficient Masked Generation](https://mlsanity.com/paper/2505.19089/) (2025)
- [Masked Generative Nested Transformers with Decode Time Scaling](https://mlsanity.com/paper/2502.00382/) (2025)
- [Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials](https://mlsanity.com/paper/2511.00833/) (2025)
- [SceneNAT: Masked Generative Modeling for Language-Guided Indoor Scene Synthesis](https://mlsanity.com/paper/2601.07218/) (2026)
- [Consistent Diffusion: Denoising Diffusion Model with Data-Consistent Training for Image Restoration](https://mlsanity.com/paper/2412.12550/) (2024)
- [Improved Masked Image Generation with Knowledge-Augmented Token Representations](https://mlsanity.com/paper/2511.12032/) (2025)
- [AdaGen: Learning Adaptive Policy for Image Synthesis](https://mlsanity.com/paper/2603.06993/) (2026)
- [DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models](https://mlsanity.com/paper/2511.21415/) (2025)
- [CSD: Content-aware Speculative Decoding for Efficient Image Generation](https://mlsanity.com/paper/2606.27829/) (2026)

## Abstract (arXiv)

The field of image synthesis is currently flourishing due to the advancements in diffusion models. While diffusion models have been successful, their computational intensity has prompted the pursuit of more efficient alternatives. As a representative work, non-autoregressive Transformers (NATs) have been recognized for their rapid generation. However, a major drawback of these models is their inferior performance compared to diffusion models. In this paper, we aim to re-evaluate the full potential of NATs by revisiting the design of their training and inference strategies. Specifically, we identify the complexities in properly configuring these strategies and indicate the possible sub-optimality in existing heuristic-driven designs. Recognizing this, we propose to go beyond existing methods by directly solving the optimal strategies in an automatic framework. The resulting method, named AutoNAT, advances the performance boundaries of NATs notably, and is able to perform comparably with the latest diffusion models at a significantly reduced inference cost. The effectiveness of AutoNAT is validated on four benchmark datasets, i.e., ImageNet-256 & 512, MS-COCO, and CC3M. Our code is available at https://github.com/LeapLabTHU/ImprovedNAT.