mlsanityData to 7 Oct 2026

Fast Training of Diffusion Models with Masked Transformers

arXiv 2306.09305 ↗First posted 15 Jun 2023cs.CV cs.AI cs.LGcited by 172

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We propose an efficient approach to train large diffusion models with masked transformers. While masked transformers have been extensively explored for representation learning, their application to generative learning is less explored in the vision domain. Our work is the first to exploit masked training to reduce the training cost of diffusion models significantly. Specifically, we randomly mask out a high proportion (e.g., 50%) of patches in diffused input images during training. For masked training, we introduce an asymmetric encoder-decoder architecture consisting of a transformer encoder that operates only on unmasked patches and a lightweight transformer decoder on full patches. To promote a long-range understanding of full patches, we add an auxiliary task of reconstructing masked patches to the denoising score matching objective that learns the score of unmasked patches. Experiments on ImageNet-256x256 and ImageNet-512x512 show that our approach achieves competitive and even better generative performance than the state-of-the-art Diffusion Transformer (DiT) model, using only around 30% of its original training time. Thus, our method shows a promising way of efficiently training large transformer-based diffusion models without sacrificing the generative performance.

Key facts. Cited by 172 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 48 of its 55 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

48 of 55 references.

Closest ancestors · 20

Same direction, newest first

  1. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models25 Apr
  2. 2023Diffusion Models as Masked Autoencoders6 Apr
  3. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  4. 2023Consistency Models2 Mar
  5. 2023I2SB: Image-to-Image Schr"odinger Bridge12 Feb
  6. 2022Scalable Diffusion Models with Transformers19 Dec
  7. 2022Fast Sampling of Diffusion Models via Operator Learning24 Nov
  8. 2022On Distillation of Guided Diffusion Models6 Oct
  9. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  10. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model16 Aug
  11. 2022Classifier-Free Diffusion Guidance26 Jul
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
8 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  3. 2021Score-based Generative Modeling in Latent Space10 Jun
  4. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  5. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  7. 2020Denoising Diffusion Implicit Models6 Oct
  8. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
16 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  4. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  5. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  6. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  7. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  8. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  9. 2022Diffusion Models for Adversarial Purificationcited by 469
  10. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  11. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  12. 2021Generating Images with Sparse Representationscited by 262
  13. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  14. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  15. 2022(Certified!!) Adversarial Robustness for Free!cited by 130
  16. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 41; 2025: 69; 2026: 51.

172 papers.

Sort:

By area:

  1. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  2. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  3. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  4. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  5. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
  6. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  7. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  8. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  9. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  10. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformerscited by 212
  11. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 138
  12. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 116
38 more
  1. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  2. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 95
  3. 2025EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modelingcited by 85
  4. 2025Latent Diffusion Model without Variational Autoencodercited by 83
  5. 2023GenTron: Diffusion Transformers for Image and Video Generationcited by 80
  6. 2023Diffusion Models Without Attentioncited by 75
  7. 2025PixelDiT: Pixel Diffusion Transformers for Image Generationcited by 75
  8. 2024FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modificationcited by 64
  9. 2024Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformercited by 61
  10. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  11. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselvescited by 50
  12. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformercited by 47
  13. 2024Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Stepcited by 47
  14. 2023Pyramid Diffusion for Fine 3D Large Scene Generationcited by 44
  15. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 42
  16. 2025DiP: Taming Diffusion Models in Pixel Spacecited by 42
  17. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  18. 2024Scaling Laws For Diffusion Transformerscited by 41
  19. 2024On Statistical Rates and Provably Efficient Criteria of Latent Diffusion Transformers (DiTs)cited by 39
  20. 2024On the Scalability of Diffusion-based Text-to-Image Generationcited by 36
  21. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  22. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 35
  23. 2025REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Trainingcited by 33
  24. 2024Ambient Diffusion Posterior Sampling: Solving Inverse Problems with Diffusion Models Trained on Corrupted Datacited by 32
  25. 2025Transition Models: Rethinking the Generative Learning Objectivecited by 32
  26. 2024Causal Diffusion Transformers for Generative Modelingcited by 31
  27. 2025STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesiscited by 31
  28. 2024A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Trainingcited by 30
  29. 2024Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budgetcited by 30
  30. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 29
  31. 2025U-REPA: Aligning Diffusion U-Nets to ViTscited by 29
  32. 2025MaskGWM: A Generalizable Driving World Model with Video Mask Reconstructioncited by 27
  33. 2025One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generationcited by 27
  34. 2024Fast, Scale-Adaptive, and Uncertainty-Aware Downscaling of Earth System Model Fields with Generative Machine Learningcited by 26
  35. 2026Self-Supervised Flow Matching for Scalable Multi-Modal Synthesiscited by 25
  36. 2024Hierarchical Patch Diffusion Models for High-Resolution Video Generationcited by 24
  37. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediatorscited by 24
  38. 2024STIV: Scalable Text and Image Conditioned Video Generationcited by 24

50 most cited of 172 All 172 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparsely Supervised Diffusion
  2. 2024Simplified and Generalized Masked Diffusion for Discrete Data
  3. 2026DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity
  4. 2022Exploring Vision Transformers as Diffusion Learners
  5. 2024Unified Auto-Encoding with Masked Diffusion
  6. 2024Diffscaler: Enhancing the Generative Prowess of Diffusion Transformers
  7. 2022Representation Learning with Diffusion Models
  8. 2026Should We Skip Diffusion?