mlsanityData to 14 Oct 2026

DiffiT: Diffusion Vision Transformers for Image Generation

arXiv 2312.02139 ↗First posted 4 Dec 2023cs.CV cs.AI cs.LGcited by 106

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models with their powerful expressivity and high sample quality have achieved State-Of-The-Art (SOTA) performance in the generative domain. The pioneering Vision Transformer (ViT) has also demonstrated strong modeling capabilities and scalability, especially for recognition tasks. In this paper, we study the effectiveness of ViTs in diffusion-based generative learning and propose a new model denoted as Diffusion Vision Transformers (DiffiT). Specifically, we propose a methodology for finegrained control of the denoising process and introduce the Time-dependant Multihead Self Attention (TMSA) mechanism. DiffiT is surprisingly effective in generating high-fidelity images with significantly better parameter efficiency. We also propose latent and image space DiffiT models and show SOTA performance on a variety of class-conditional and unconditional synthesis tasks at different resolutions. The Latent DiffiT model achieves a new SOTA FID score of 1.73 on ImageNet256 dataset while having 19.85%, 16.88% less parameters than other Transformer-based diffusion models such as MDT and DiT,respectively. Code: https://github.com/NVlabs/DiffiT

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 72 of its 88 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

72 of 88 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  2. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  3. 2022Scalable Diffusion Models with Transformers19 Dec
  4. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model16 Aug
  5. 2022Improving Diffusion Model Efficiency Through Patching9 Jul
  6. 2022gDDIM: Generalized denoising diffusion implicit models11 Jun
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  8. 2022Maximum Likelihood Training of Implicit Nonlinear Diffusion Models27 May
  9. 2022Perception Prioritized Training of Diffusion Models1 Apr
  10. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  11. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusion14 Dec
  12. 2021Score-based Generative Modeling in Latent Space10 Jun
5 more
  1. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  2. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  4. 2020Denoising Diffusion Implicit Models6 Oct
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 55

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2016Layer Normalizationcited by 5,669
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2013Auto-Encoding Variational Bayescited by 4,053
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
38 more
  1. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  5. 2018Group Normalizationcited by 1,524
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  7. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  8. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  9. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  10. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  11. 2018Self-Attention with Relative Position Representationscited by 1,120
  12. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  13. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  14. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  15. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  16. 2022Denoising Diffusion Restoration Modelscited by 963
  17. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  18. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 939
  19. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  20. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  21. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  22. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  23. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 647
  24. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  25. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 582
  26. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  27. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  28. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 516
  29. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 492
  30. 2022Diffusion Models for Adversarial Purificationcited by 476
  31. 2022Blended Latent Diffusioncited by 432
  32. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  33. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  34. 2021Generative Adversarial Transformerscited by 120
  35. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  36. 2021D2C: Diffusion-Denoising Models for Few-shot Conditional Generationcited by 96
  37. 2019AutoGAN: Neural Architecture Search for Generative Adversarial Networkscited by 94
  38. 2020A Contrastive Learning Approach for Training Variational Autoencoder Priorscited by 66

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 37; 2025: 49; 2026: 19.

106 papers.

Sort:

By area:

  1. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  2. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  4. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  5. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 366
  6. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 329
  7. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  8. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 283
  9. 2024Consistency Models Made Easycited by 144
  10. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 115
  11. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  12. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 102
38 more
  1. 2023Return of Unconditional Generation: A Self-supervised Representation Generation Methodcited by 98
  2. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 89
  3. 2025Align Your Flow: Scaling Continuous-Time Flow Map Distillationcited by 87
  4. 2024FiT: Flexible Vision Transformer for Diffusion Modelcited by 85
  5. 2024Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusioncited by 75
  6. 2024Dynamic Diffusion Transformercited by 62
  7. 2025SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restorationcited by 62
  8. 2024U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformerscited by 54
  9. 2024Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Stepcited by 47
  10. 2024Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalizationcited by 36
  11. 2024Language-Guided Image Tokenization for Generationcited by 36
  12. 2025Transition Models: Rethinking the Generative Learning Objectivecited by 34
  13. 2025U-REPA: Aligning Diffusion U-Nets to ViTscited by 32
  14. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 26
  15. 2024SITCOM: Step-wise Triple-Consistent Diffusion Sampling for Inverse Problemscited by 24
  16. 2024Beyond Deepfake Images: Detecting AI-Generated Videoscited by 22
  17. 2024Probabilistic Emulation of a Global Climate Model with Spherical DYffusioncited by 21
  18. 2024Effective Diffusion Transformer Architecture for Image Super-Resolutioncited by 21
  19. 2024Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Modelscited by 21
  20. 2025QuantCache: Adaptive Importance-Guided Quantization with Hierarchical Latent and Layer Caching for Video Generationcited by 21
  21. 2025Decoupled MeanFlow: Turning Flow Models into Flow Maps for Accelerated Samplingcited by 20
  22. 2025UniVG: A Generalist Diffusion Model for Unified Image Generation and Editingcited by 19
  23. 2025Guiding a Diffusion Transformer with the Internal Dynamics of Itselfcited by 16
  24. 2024FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Modelcited by 15
  25. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  26. 2024Denoising with a Joint-Embedding Predictive Architecturecited by 14
  27. 2024DiC: Rethinking Conv3x3 Designs in Diffusion Modelscited by 14
  28. 2024High Fidelity Text-Guided Music Editing via Single-Stage Flow Matchingcited by 13
  29. 2025Effective and Efficient Masked Image Generation Modelscited by 13
  30. 2025Learning Diffusion Models with Flexible Representation Guidancecited by 13
  31. 2025FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Computecited by 12
  32. 2025SARA: Structural and Adversarial Representation Alignment for Training-efficient Diffusion Modelscited by 11
  33. 2025Attention (as Discrete-Time Markov) Chainscited by 11
  34. 2025BWCache: Accelerating Video Diffusion Transformers through Block-Wise Cachingcited by 11
  35. 2025Deeply Supervised Flow-Based Generative Modelscited by 9
  36. 2025Autoregressive Distillation of Diffusion Transformerscited by 9
  37. 2025Score-based Self-supervised MRI Denoisingcited by 9
  38. 2025LayerSync: Self-aligning Intermediate Layerscited by 9

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffscaler: Enhancing the Generative Prowess of Diffusion Transformers
  2. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer
  3. 2025DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling
  4. 2023DiffI2I: Efficient Diffusion Model for Image-to-Image Translation
  5. 2022Exploring Vision Transformers as Diffusion Learners
  6. 2022All are Worth Words: A ViT Backbone for Diffusion Models
  7. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generation
  8. 2025DiT4SR: Taming Diffusion Transformer for Real-World Image Super-Resolution