mlsanityData to 7 Oct 2026

Scalable Diffusion Models with Transformers

arXiv 2212.09748 ↗First posted 19 Dec 2022cs.CV cs.LGcited by 5,931

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

We explore a new class of diffusion models based on the transformer architecture. We train latent diffusion models of images, replacing the commonly-used U-Net backbone with a transformer that operates on latent patches. We analyze the scalability of our Diffusion Transformers (DiTs) through the lens of forward pass complexity as measured by Gflops. We find that DiTs with higher Gflops -- through increased transformer depth/width or increased number of input tokens -- consistently have lower FID. In addition to possessing good scalability properties, our largest DiT-XL/2 models outperform all prior diffusion models on the class-conditional ImageNet 512x512 and 256x256 benchmarks, achieving a state-of-the-art FID of 2.27 on the latter.

Key facts. Cited by 5,931 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 51 of its 68 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

51 of 68 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022MaskGIT: Masked Generative Image Transformer8 Feb
  2. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale22 Oct
  4. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
35 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  13. 2016Improved Techniques for Training GANscited by 2,938
  14. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  15. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  16. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  17. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  18. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  19. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  20. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  21. 2021Scaling Vision Transformerscited by 993
  22. 2020Designing Network Design Spacescited by 954
  23. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  24. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  25. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  26. 2018Image Transformercited by 770
  27. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  28. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  29. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  30. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  31. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  32. 2021Early Convolutions Help Transformers See Bettercited by 395
  33. 2021Generating Images with Sparse Representationscited by 262
  34. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  35. 2019On Network Design Spaces for Visual Recognitioncited by 68

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 195; 2024: 1018; 2025: 2328; 2026: 2388.

5,931 papers.

Sort:

By area:

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  5. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  6. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  8. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  9. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  10. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  11. 2025Large Language Diffusion Modelscited by 872
  12. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
38 more
  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  3. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  4. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  5. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  6. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  7. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  8. 2023Adversarial Diffusion Distillationcited by 741
  9. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  10. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  11. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  12. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  13. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  14. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  15. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  16. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
  17. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  18. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  19. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  20. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  21. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  22. 2025Mean Flows for One-step Generative Modelingcited by 524
  23. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  24. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  25. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  26. 2025VACE: All-in-One Video Creation and Editingcited by 467
  27. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  28. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  29. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  30. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  31. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  32. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  33. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  34. 2024OmniGen: Unified Image Generationcited by 388
  35. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  36. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  37. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  38. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376

50 most cited of 5,931 All 5,931 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Exploring Transformer Backbones for Image Diffusion Models
  2. 2026Post-Training Pruning for Diffusion Transformers
  3. 2024Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study
  4. 2023Neural Diffusion Models
  5. 2024On Statistical Rates of Conditional Diffusion Transformers: Approximation, Estimation and Minimax Optimality
  6. 2024On Statistical Rates and Provably Efficient Criteria of Latent Diffusion Transformers (DiTs)
  7. 2025Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations