# On the Scalability of Diffusion-based Text-to-Image Generation

arXiv: https://arxiv.org/abs/2404.02883 · HTML: https://mlsanity.com/paper/2404.02883/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 34 of its 40 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [All are Worth Words: A ViT Backbone for Diffusion Models](https://mlsanity.com/paper/2209.12152/) (2022)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [Simple diffusion: End-to-end diffusion for high resolution images](https://mlsanity.com/paper/2301.11093/) (2023)
- [Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack](https://mlsanity.com/paper/2309.15807/) (2023)

## Built on by (most cited first, 30 of 36)

- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer](https://mlsanity.com/paper/2501.18427/) (2025)
- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [Scaling Laws For Diffusion Transformers](https://mlsanity.com/paper/2410.08184/) (2024)
- [Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space](https://mlsanity.com/paper/2406.19370/) (2024)
- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Training](https://mlsanity.com/paper/2412.09619/) (2024)
- [Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation](https://mlsanity.com/paper/2605.28091/) (2026)
- [GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI](https://mlsanity.com/paper/2511.15658/) (2025)
- [Towards Precise Scaling Laws for Video Diffusion Transformers](https://mlsanity.com/paper/2411.17470/) (2024)
- [FLUX that Plays Music](https://mlsanity.com/paper/2409.00587/) (2024)
- [Scaling Properties of Diffusion Models for Perceptual Tasks](https://mlsanity.com/paper/2411.08034/) (2024)
- [TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models](https://mlsanity.com/paper/2406.05814/) (2024)
- [Erasing More Than Intended? How Concept Erasure Degrades the Generation of Non-Target Concepts](https://mlsanity.com/paper/2501.09833/) (2025)
- [AERIS: Argonne Earth Systems Model for Reliable and Skillful Predictions](https://mlsanity.com/paper/2509.13523/) (2025)
- [Quality-aware Masked Diffusion Transformer for Enhanced Music Generation](https://mlsanity.com/paper/2405.15863/) (2024)
- [Scaling Diffusion Transformers Efficiently via μP](https://mlsanity.com/paper/2505.15270/) (2025)
- [YaART: Yet Another ART Rendering Technology](https://mlsanity.com/paper/2404.05666/) (2024)
- [Swing-by Dynamics in Concept Learning and Compositional Generalization](https://mlsanity.com/paper/2410.08309/) (2024)
- [AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation](https://mlsanity.com/paper/2411.04967/) (2024)
- [Efficient Scaling of Diffusion Transformers for Text-to-Image Generation](https://mlsanity.com/paper/2412.12391/) (2024)
- [FlexCache: Flexible Approximate Cache System for Video Diffusion](https://mlsanity.com/paper/2501.04012/) (2024)
- [Diagnosing and Improving Diffusion Models by Estimating the Optimal Loss Value](https://mlsanity.com/paper/2506.13763/) (2025)
- [How to Train your Text-to-Image Model: Evaluating Design Choices for Synthetic Training Captions](https://mlsanity.com/paper/2506.16679/) (2025)
- [World-To-Image: Grounding Text-to-Image Generation with Agent-Driven World Knowledge](https://mlsanity.com/paper/2510.04201/) (2025)
- [MuDiT & MuSiT: Alignment with Colloquial Expression in Description-to-Song Generation](https://mlsanity.com/paper/2407.03188/) (2024)
- [The Seriality Gap in Video Diffusion Models](https://mlsanity.com/paper/2607.13031/) (2026)
- [From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation](https://mlsanity.com/paper/2608.18076/) (2026)
- [One Small Step in Latent, One Giant Leap for Pixels: Fast Latent Upscale Adapter for Your Diffusion Models](https://mlsanity.com/paper/2511.10629/) (2025)
- [Abra: Scaling Diffusion Image Training](https://mlsanity.com/paper/2608.17286/) (2026)

## Abstract (arXiv)

Scaling up model and data size has been quite successful for the evolution of LLMs. However, the scaling law for the diffusion based text-to-image (T2I) models is not fully explored. It is also unclear how to efficiently scale the model for better performance at reduced cost. The different training settings and expensive training cost make a fair model comparison extremely difficult. In this work, we empirically study the scaling properties of diffusion based T2I models by performing extensive and rigours ablations on scaling both denoising backbones and training set, including training scaled UNet and Transformer variants ranging from 0.4B to 4B parameters on datasets upto 600M images. For model scaling, we find the location and amount of cross attention distinguishes the performance of existing UNet designs. And increasing the transformer blocks is more parameter-efficient for improving text-image alignment than increasing channel numbers. We then identify an efficient UNet variant, which is 45% smaller and 28% faster than SDXL's UNet. On the data scaling side, we show the quality and diversity of the training set matters more than simply dataset size. Increasing caption density and diversity improves text-image alignment performance and the learning efficiency. Finally, we provide scaling functions to predict the text-image alignment performance as functions of the scale of model size, compute and dataset size.