mlsanityData to 7 Oct 2026

Efficient Scaling of Diffusion Transformers for Text-to-Image Generation

arXiv 2412.12391 ↗First posted 16 Dec 2024cs.CV cs.CL cs.LGcited by 5

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We empirically study the scaling properties of various Diffusion Transformers (DiTs) for text-to-image generation by performing extensive and rigorous ablations, including training scaled DiTs ranging from 0.3B upto 8B parameters on datasets up to 600M images. We find that U-ViT, a pure self-attention based DiT model provides a simpler design and scales more effectively in comparison with cross-attention based DiT variants, which allows straightforward expansion for extra conditions and other modalities. We identify a 2.3B U-ViT model can get better performance than SDXL UNet and other DiT variants in controlled setting. On the data scaling side, we investigate how increasing dataset size and enhanced long caption improve the text-image alignment performance and the learning efficiency.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 39 of its 43 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

39 of 43 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT5 Jun
  2. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers9 May
  3. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation7 Mar
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  5. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers21 Jan
  6. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis30 Sep
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  8. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  9. 2023On the Importance of Noise Scheduling for Diffusion Models26 Jan
  10. 2022Scalable Diffusion Models with Transformers19 Dec
  11. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  12. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
2 more
  1. 2020Denoising Diffusion Implicit Models6 Oct
  2. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
13 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  4. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  5. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  6. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  7. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  8. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  9. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  10. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  11. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusioncited by 200
  12. 2024On the Scalability of Diffusion-based Text-to-Image Generationcited by 36
  13. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Modelscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Scaling Diffusion Transformers Efficiently via μPcited by 6
  2. 2026Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestepcited by 2
  3. 2025Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformercited by 1
  4. 2026Abra: Scaling Diffusion Image Trainingcited by 1
  5. 2026Importance-Aware OBS Pruning for Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DiT-Air: Revisiting the Efficiency of Diffusion Model Architecture Design in Text to Image Generation
  2. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing
  3. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflection
  4. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
  5. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer
  6. 2024In-Context LoRA for Diffusion Transformers
  7. 2024Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis
  8. 2023Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis