mlsanityData to 7 Oct 2026

Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis

arXiv 2406.05478 ↗First posted 8 Jun 2024cs.CV cs.AIcited by 22

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

The field of image synthesis is currently flourishing due to the advancements in diffusion models. While diffusion models have been successful, their computational intensity has prompted the pursuit of more efficient alternatives. As a representative work, non-autoregressive Transformers (NATs) have been recognized for their rapid generation. However, a major drawback of these models is their inferior performance compared to diffusion models. In this paper, we aim to re-evaluate the full potential of NATs by revisiting the design of their training and inference strategies. Specifically, we identify the complexities in properly configuring these strategies and indicate the possible sub-optimality in existing heuristic-driven designs. Recognizing this, we propose to go beyond existing methods by directly solving the optimal strategies in an automatic framework. The resulting method, named AutoNAT, advances the performance boundaries of NATs notably, and is able to perform comparably with the latest diffusion models at a significantly reduced inference cost. The effectiveness of AutoNAT is validated on four benchmark datasets, i.e., ImageNet-256 & 512, MS-COCO, and CC3M. Our code is available at https://github.com/LeapLabTHU/ImprovedNAT.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 46 of its 51 references that are in the map. First posted on arXiv 8 Jun 2024.

Builds on

46 of 51 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023One-step Diffusion with Distribution Matching Distillation30 Nov
  2. 2023Adversarial Diffusion Distillation28 Nov
  3. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference6 Oct
  4. 2023AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration19 Sep
  5. 2022Scalable Diffusion Models with Transformers19 Dec
  6. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  7. 2022Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis29 Aug
  8. 2022Classifier-Free Diffusion Guidance26 Jul
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  10. 2022Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality11 Feb
  11. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  12. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesis29 Nov
3 more
  1. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  2. 2020Denoising Diffusion Implicit Models6 Oct
  3. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
19 more
  1. 2021Convex optimizationcited by 2,479
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  6. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  7. 2016An overview of gradient descent optimization algorithmscited by 1,021
  8. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  9. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  10. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  11. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  12. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  13. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 205
  14. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  15. 2016Neural Photo Editing with Introspective Adversarial Networkscited by 109
  16. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  17. 2022Improved Masked Image Generation with Token-Criticcited by 63
  18. 2022Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformercited by 30
  19. 2023StraIT: Non-autoregressive Generation with Stratified Image Transformercited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 10; 2026: 4.

22 papers.

Sort:

By area:

  1. 2024Bridging the Divide: Reconsidering Softmax and Linear Attentioncited by 48
  2. 2024TinyFusion: Diffusion Transformers Learned Shallowcited by 44
  3. 2025Halton Scheduler For Masked Generative Image Transformercited by 30
  4. 2024Real-world Image Dehazing with Coherence-based Pseudo Labeling and Cooperative Unfolding Networkcited by 26
  5. 2024Customize Your Visual Autoregressive Recipe with Set Autoregressive Modelingcited by 21
  6. 2024AdaNAT: Exploring Adaptive Policy for Token-Based Image Generationcited by 13
  7. 2025MagicTryOn: Harnessing Diffusion Transformer for Garment-Preserving Video Virtual Try-oncited by 12
  8. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 11
  9. 2025Autoregressive Distillation of Diffusion Transformerscited by 9
  10. 2024ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesiscited by 8
  11. 2025CODA: Repurposing Continuous VAEs for Discrete Tokenizationcited by 8
  12. 2025Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivotscited by 8
10 more
  1. 2024MaskMamba: A Hybrid Mamba-Transformer Model for Masked Image Generationcited by 7
  2. 2025Plug-and-Play Context Feature Reuse for Efficient Masked Generationcited by 5
  3. 2025Masked Generative Nested Transformers with Decode Time Scalingcited by 4
  4. 2025Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentialscited by 4
  5. 2026SceneNAT: Masked Generative Modeling for Language-Guided Indoor Scene Synthesiscited by 2
  6. 2024Consistent Diffusion: Denoising Diffusion Model with Data-Consistent Training for Image Restorationcited by 1
  7. 2025Improved Masked Image Generation with Knowledge-Augmented Token Representationscited by 1
  8. 2025DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Modelscited by 0
  9. 2026AdaGen: Learning Adaptive Policy for Image Synthesiscited by 0
  10. 2026CSD: Content-aware Speculative Decoding for Efficient Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation
  2. 2025FREE: Uncertainty-Aware Autoregression for Parallel Diffusion Transformers
  3. 2022Exploring Transformer Backbones for Image Diffusion Models
  4. 2026RT-Lynx: Putting GEMM Sparsity in the Right Place for Diffusion Models
  5. 2026Just-in-Time: Training-Free Spatial Acceleration for Diffusion Transformers
  6. 2025Universal Approximation of Visual Autoregressive Transformers
  7. 2024DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation
  8. 2026From Sketch to Fresco: Efficient Diffusion Transformer with Progressive Resolution