mlsanityData to 14 Oct 2026

Scaling Laws For Diffusion Transformers

arXiv 2410.08184 ↗First posted 10 Oct 2024cs.CVcited by 44

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion transformers (DiT) have already achieved appealing synthesis and scaling properties in content recreation, e.g., image and video generation. However, scaling laws of DiT are less explored, which usually offer precise predictions regarding optimal model size and data requirements given a specific compute budget. Therefore, experiments across a broad range of compute budgets, from 1e17 to 6e18 FLOPs are conducted to confirm the existence of scaling laws in DiT for the first time. Concretely, the loss of pretraining DiT also follows a power-law relationship with the involved compute. Based on the scaling law, we can not only determine the optimal model size and required data but also accurately predict the text-to-image generation loss given a model with 1B parameters and a compute budget of 1e21 FLOPs. Additionally, we also demonstrate that the trend of pre-training loss matches the generation performances (e.g., FID), even across various datasets, which complements the mapping from compute to synthesis quality and thus provides a predictable benchmark that assesses model performance and data quality at a reduced cost.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 53 of its 63 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

53 of 63 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024On Statistical Rates and Provably Efficient Criteria of Latent Diffusion Transformers (DiTs)1 Jul
  2. 2024Bigger is not Always Better: Scaling Properties of Latent Diffusion Models1 Apr
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  4. 2024FiT: Flexible Vision Transformer for Diffusion Model19 Feb
  5. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  6. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis30 Sep
  7. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  8. 2022Scalable Diffusion Models with Transformers19 Dec
  9. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  10. 2022Classifier-Free Diffusion Guidance26 Jul
  11. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  12. 2021Variational Diffusion Models1 Jul
5 more
  1. 2021Score-based Generative Modeling in Latent Space10 Jun
  2. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  3. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2022Flow Matching for Generative Modelingcited by 5,702
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
24 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  7. 2016Density estimation using Real NVPcited by 2,042
  8. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  11. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  12. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  13. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 939
  14. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  15. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  16. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  17. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 594
  18. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  19. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  20. 2021Projected GANs Converge Fastercited by 208
  21. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
  22. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  23. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 56
  24. 2024On the Scalability of Diffusion-based Text-to-Image Generationcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 21; 2026: 18.

44 papers.

Sort:

By area:

  1. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  2. 2025Seaweed-7B: Cost-Effective Training of Video Generation Foundation Modelcited by 82
  3. 2024xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelismcited by 56
  4. 2024ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Modelcited by 53
  5. 2025Scaling Laws for Optimal Data Mixturescited by 52
  6. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  7. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 43
  8. 2025TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Trainingcited by 21
  9. 2024Towards Precise Scaling Laws for Video Diffusion Transformerscited by 20
  10. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  11. 2025Normalized Attention Guidance: Universal Negative Guidance for Diffusion Modelscited by 13
  12. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generationcited by 10
32 more
  1. 2025How far can we go with ImageNet for Text-to-Image generation?cited by 10
  2. 2025DiT-Air: Revisiting the Efficiency of Diffusion Model Architecture Design in Text to Image Generationcited by 10
  3. 2024Turbo-GS: Accelerating 3D Gaussian Fitting for High-Quality Radiance Fieldscited by 8
  4. 2025AERIS: Argonne Earth Systems Model for Reliable and Skillful Predictionscited by 8
  5. 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budgetcited by 6
  6. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelinescited by 5
  7. 2025Diagnosing and Improving Diffusion Models by Estimating the Optimal Loss Valuecited by 5
  8. 2025Scalable GANs with Transformerscited by 5
  9. 2025T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoderscited by 4
  10. 2025Scaling Laws are Redundancy Lawscited by 3
  11. 2026Normalizing Flows with Iterative Denoisingcited by 3
  12. 2026YoCausal: How Far is Video Generation from World Model? A Causality Perspectivecited by 3
  13. 2026Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformerscited by 3
  14. 2026Unified Neural Scaling Lawscited by 2
  15. 2026The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluationcited by 2
  16. 2026Scaling Properties of Text Conditioning in Visual Generationcited by 2
  17. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementationcited by 1
  18. 2025AMix-1: A Pathway to Test-Time Scalable Protein Foundation Modelcited by 1
  19. 2026Scaling Properties of Continuous Diffusion Spoken Language Modelscited by 1
  20. 2026Complexity-Balanced Diffusion Splittingcited by 1
  21. 2026Abra: Scaling Diffusion Image Trainingcited by 1
  22. 2026Scaling Muon for Diffusion Transformerscited by 1
  23. 2025Training Video Foundation Models with NVIDIA NeMocited by 0
  24. 2025Bootstrapping Diffusion: Diffusion Model Training Leveraging Partial and Corrupted Datacited by 0
  25. 2025Iterative Inference-time Scaling with Adaptive Frequency Steering for Image Super-Resolutioncited by 0
  26. 2026Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fitscited by 0
  27. 2026Motif-Video 2B: Technical Reportcited by 0
  28. 2026Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspectivecited by 0
  29. 2026Explicit Critic Guidance for Aligning Diffusion Modelscited by 0
  30. 2026How Far Can 5,500 Hours of Driving Take You? A Scaling Law Analysis of Video Diffusion Modelscited by 0
  31. 2026Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCachecited by 0
  32. 2026Scaling Video Generation for Reasoning: At What Cost?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study
  2. 2024FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification
  3. 2024Dynamic Diffusion Transformer
  4. 2024Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers
  5. 2026Elastic Diffusion Transformer
  6. 2026DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers
  7. 2025Designing Parameter and Compute Efficient Diffusion Transformers using Distillation
  8. 2025Scaling Diffusion Transformers Efficiently via μP