mlsanityData to 7 Oct 2026

All are Worth Words: A ViT Backbone for Diffusion Models

arXiv 2209.12152 ↗First posted 25 Sep 2022cs.CV cs.AI cs.LGcited by 498

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Vision transformers (ViT) have shown promise in various vision tasks while the U-Net based on a convolutional neural network (CNN) remains dominant in diffusion models. We design a simple and general ViT-based architecture (named U-ViT) for image generation with diffusion models. U-ViT is characterized by treating all inputs including the time, condition and noisy image patches as tokens and employing long skip connections between shallow and deep layers. We evaluate U-ViT in unconditional and class-conditional image generation, as well as text-to-image generation tasks, where U-ViT is comparable if not superior to a CNN-based U-Net of a similar size. In particular, latent diffusion models with U-ViT achieve record-breaking FID scores of 2.29 in class-conditional image generation on ImageNet 256x256, and 5.48 in text-to-image generation on MS-COCO, among methods without accessing large external datasets during the training of generative models. Our results suggest that, for diffusion-based image modeling, the long skip connection is crucial while the down-sampling and up-sampling operators in CNN-based U-Net are not always necessary. We believe that U-ViT can provide insights for future research on backbones in diffusion models and benefit generative modeling on large scale cross-modality datasets.

Key facts. Cited by 498 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 72 of its 89 references that are in the map. First posted on arXiv 25 Sep 2022.

Builds on

72 of 89 references.

Closest ancestors · 29

Same direction, newest first

  1. 2022Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis29 Aug
  2. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model16 Aug
  3. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning8 Aug
  4. 2022Classifier-Free Diffusion Guidance26 Jul
  5. 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations14 Jul
  6. 2022Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching16 Jun
  7. 2022Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models15 Jun
  8. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  10. 2022Improved Vector Quantized Diffusion Models31 May
  11. 2022Generating High Fidelity Data from Low-density Regions using Diffusion Models31 Mar
  12. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models17 Jan
17 more
  1. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  2. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANs15 Dec
  3. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusion14 Dec
  4. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesis29 Nov
  5. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models6 Aug
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  7. 2021Variational Diffusion Models1 Jul
  8. 2021Score-based Generative Modeling in Latent Space10 Jun
  9. 2021Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation10 Jun
  10. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  11. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  12. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  13. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  14. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  15. 2020Denoising Diffusion Implicit Models6 Oct
  16. 2020Denoising Diffusion Probabilistic Models19 Jun
  17. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
31 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2016Improved Techniques for Training GANscited by 2,938
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  5. 2022Video Diffusion Modelscited by 2,280
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  9. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  11. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  12. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  13. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  14. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  15. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  16. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  17. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  18. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  19. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  20. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  21. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  22. 2021Early Convolutions Help Transformers See Bettercited by 395
  23. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  24. 2021Generating Images with Sparse Representationscited by 262
  25. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  26. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  27. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  28. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  29. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  30. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  31. 2021Zero-Shot Translation using Diffusion Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 31; 2024: 177; 2025: 184; 2026: 103.

498 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  2. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  3. 2025Large Language Diffusion Modelscited by 872
  4. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  5. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  6. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  7. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  8. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  9. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  10. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  11. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  12. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
38 more
  1. 2023Text-to-3D using Gaussian Splattingcited by 299
  2. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  3. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  4. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  5. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 293
  6. 2024Guiding a Diffusion Model with a Bad Version of Itselfcited by 286
  7. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  8. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 242
  9. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  10. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
  11. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 209
  12. 2022Fast Sampling of Diffusion Models via Operator Learningcited by 173
  13. 2023Fast Training of Diffusion Models with Masked Transformerscited by 172
  14. 2025History-Guided Video Diffusioncited by 165
  15. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  16. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
  17. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Modelscited by 159
  18. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  19. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 158
  20. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  21. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  22. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
  23. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 138
  24. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  25. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  26. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  27. 2024Randomized Autoregressive Visual Generationcited by 125
  28. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 116
  29. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  30. 2022Latent Diffusion for Language Generationcited by 110
  31. 2023DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generationcited by 110
  32. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  33. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 108
  34. 2025Inductive Moment Matchingcited by 107
  35. 2025DDT: Decoupled Diffusion Transformercited by 100
  36. 2024MaskBit: Embedding-free Image Generation via Bit Tokenscited by 96
  37. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 95
  38. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 93

50 most cited of 498 All 498 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Exploring Vision Transformers as Diffusion Learners
  2. 2023DiffiT: Diffusion Vision Transformers for Image Generation
  3. 2026ViT-5: Vision Transformers for The Mid-2020s
  4. 2022Exploring Transformer Backbones for Image Diffusion Models
  5. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models
  6. 2023Diffusion Models Beat GANs on Image Classification
  7. 2022Swinv2-Imagen: Hierarchical Vision Transformer Diffusion Models for Text-to-Image Generation
  8. 2023U-DiT TTS: U-Diffusion Vision Transformer for Text-to-Speech