mlsanityData to 7 Oct 2026

Exploring Vision Transformers as Diffusion Learners

arXiv 2212.13771 ↗First posted 28 Dec 2022cs.CVcited by 10

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Score-based diffusion models have captured widespread attention and funded fast progress of recent vision generative tasks. In this paper, we focus on diffusion model backbone which has been much neglected before. We systematically explore vision Transformers as diffusion learners for various generative tasks. With our improvements the performance of vanilla ViT-based backbone (IU-ViT) is boosted to be on par with traditional U-Net-based methods. We further provide a hypothesis on the implication of disentangling the generative backbone as an encoder-decoder structure and show proof-of-concept experiments verifying the effectiveness of a stronger encoder for generative tasks with ASymmetriC ENcoder Decoder (ASCEND). Our improvements achieve competitive results on CIFAR-10, CelebA, LSUN, CUB Bird and large-resolution text-to-image tasks. To the best of our knowledge, we are the first to successfully train a single diffusion model on text-to-image task beyond 64x64 resolution. We hope this will motivate people to rethink the modeling choices and the training pipelines for diffusion-based generative models.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 59 of its 67 references that are in the map. First posted on arXiv 28 Dec 2022.

Builds on

59 of 67 references.

Closest ancestors · 14

Same direction, newest first

  1. 2022Scalable Diffusion Models with Transformers19 Dec
  2. 2022On Distillation of Guided Diffusion Models6 Oct
  3. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model16 Aug
  4. 2022Classifier-Free Diffusion Guidance26 Jul
  5. 2022Perception Prioritized Training of Diffusion Models1 Apr
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  7. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANs15 Dec
  8. 2021Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation10 Jun
  9. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  10. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  12. 2020Denoising Diffusion Implicit Models6 Oct
2 more
  1. 2020Denoising Diffusion Probabilistic Models19 Jun
  2. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
33 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  6. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  8. 2018Group Normalizationcited by 1,514
  9. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  10. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  11. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  12. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  13. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  14. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  15. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  16. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  17. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  18. 2021Multiscale Vision Transformerscited by 896
  19. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  20. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  21. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  22. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 486
  23. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  24. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  25. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  26. 2017GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibriumcited by 208
  27. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  28. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 161
  29. 2021Pros and Cons of GAN Evaluation Measures: New Developmentscited by 121
  30. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  31. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  32. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-trainingcited by 67
  33. 2022Swinv2-Imagen: Hierarchical Vision Transformer Diffusion Models for Text-to-Image Generationcited by 7

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 2; 2025: 1.

10 papers.

  1. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
  2. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 128
  3. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  4. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  5. 2024U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformerscited by 51
  6. 2023Multi-Architecture Multi-Expert Diffusion Modelscited by 34
  7. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  8. 2023Masked Diffusion Models Are Fast Distribution Learnerscited by 7
  9. 2024Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoisingcited by 5
  10. 2025Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applicationscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2022All are Worth Words: A ViT Backbone for Diffusion Models
  2. 2024Diffscaler: Enhancing the Generative Prowess of Diffusion Transformers
  3. 2024LaVin-DiT: Large Vision Diffusion Transformer
  4. 2024Diffusion Models in Low-Level Vision: A Survey
  5. 2026Reviving ConvNeXt for Efficient Convolutional Diffusion Models
  6. 2023DiffiT: Diffusion Vision Transformers for Image Generation
  7. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception
  8. 2023Fast Training of Diffusion Models with Masked Transformers