mlsanityData to 7 Oct 2026

Scaling Laws of Synthetic Images for Model Training ... for Now

arXiv 2312.04567 ↗First posted 7 Dec 2023cs.CVcited by 107

Not on the map. Why?

Abstract (arXiv)

Recent significant advances in text-to-image models unlock the possibility of training vision systems using synthetic images, potentially overcoming the difficulty of collecting curated data at scale. It is unclear, however, how these models behave at scale, as more synthetic data is added to the training set. In this paper we study the scaling laws of synthetic images generated by state of the art text-to-image models, for the training of supervised models: image classifiers with label supervision, and CLIP with language supervision. We identify several factors, including text prompts, classifier-free guidance scale, and types of text-to-image models, that significantly affect scaling behavior. After tuning these factors, we observe that synthetic images demonstrate a scaling trend similar to, but slightly less effective than, real images in CLIP training, while they significantly underperform in scaling when training supervised image classifiers. Our analysis indicates that the main reason for this underperformance is the inability of off-the-shelf text-to-image models to generate certain concepts, a limitation that significantly impairs the training of image classifiers. Our findings also suggest that scaling synthetic data can be particularly effective in scenarios such as: (1) when there is a limited supply of real images for a supervised problem (e.g., fewer than 0.5 million images in ImageNet), (2) when the evaluation dataset diverges significantly from the training data, indicating the out-of-distribution scenario, or (3) when synthetic data is used in conjunction with real images, as demonstrated in the training of CLIP models.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 67 of its 95 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

67 of 95 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  13. 2019Natural Adversarial Examplescited by 1,401
  14. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  15. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  16. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  17. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  18. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  19. 2021Scaling Vision Transformerscited by 993
  20. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  21. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  22. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  23. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  24. 2021ImageNet-21K Pretraining for the Massescited by 607
  25. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  26. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  27. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  28. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  29. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  30. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  31. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  32. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  33. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  34. 2021Scaling Laws for Transfercited by 265
  35. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  36. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  37. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 171
  38. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 44; 2025: 38; 2026: 24.

107 papers.

Sort:

By area:

  1. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  2. 2024Scaling Laws for Precisioncited by 92
  3. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  4. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  5. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  6. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
  7. 2024Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Imagescited by 41
  8. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  9. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  10. 2024Learning General-Purpose Biomedical Volume Representations using Randomized Synthesiscited by 27
  11. 2024Scaling laws for learning with real and surrogate datacited by 26
  12. 2024Synth2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddingscited by 18
38 more
  1. 2025Improving the Scaling Laws of Synthetic Data with Deliberate Practicecited by 17
  2. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  3. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  4. 2024GLASS: Guided Latent Slot Diffusion for Object-Centric Learningcited by 14
  5. 2024What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insightscited by 13
  6. 2025Scaling Tumor Segmentation: Best Lessons from Real and Synthetic Datacited by 13
  7. 2024Advancing Fine-Grained Classification by Structure and Subject Preserving Augmentationcited by 12
  8. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  9. 2025A Large Scale Analysis of Gender Biases in Text-to-Image Generative Modelscited by 12
  10. 2024Slight Corruption in Pre-training Data Makes Better Diffusion Modelscited by 9
  11. 2024Personalized Representation from Personalized Generationcited by 9
  12. 2025ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentationcited by 9
  13. 2024BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Modelscited by 7
  14. 2024Onboard Satellite Image Classification for Earth Observation: A Comparative Study of ViT Modelscited by 7
  15. 2025Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problemscited by 7
  16. 2024SynFER: Towards Boosting Facial Expression Recognition with Synthetic Datacited by 6
  17. 2024A Little Human Data Goes A Long Waycited by 6
  18. 2024Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?cited by 6
  19. 2025Generative Data Refinement: Just Ask for Better Datacited by 6
  20. 2025The Intricate Dance of Prompt Complexity, Quality, Diversity, and Consistency in T2I Modelscited by 6
  21. 2026When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generatorscited by 6
  22. 2024Scaling Laws for Galaxy Imagescited by 5
  23. 2024Are Images Indistinguishable to Humans Also Indistinguishable to Classifiers?cited by 5
  24. 2024DALDA: Data Augmentation Leveraging Diffusion Model and LLM with Adaptive Guidance Scalingcited by 5
  25. 2025High-dimensional Analysis of Synthetic Data Selectioncited by 5
  26. 2025S3OD: Towards Generalizable Salient Object Detection with Synthetic Datacited by 5
  27. 2025Training Data Provenance Verification: Did Your Model Use Synthetic Data from My Generative Model for Training?cited by 4
  28. 2025Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Datacited by 4
  29. 2024DiffuLT: How to Make Diffusion Model Useful for Long-tail Recognitioncited by 3
  30. 2024Weak-to-Strong Compositional Learning from Generative Models for Language-based Object Detectioncited by 3
  31. 2024Exploring the Potential of Synthetic Data to Replace Real Datacited by 3
  32. 2024Synthetica: Large Scale Synthetic Data for Robot Perceptioncited by 3
  33. 2024Exploring the Spectrum of Visio-Linguistic Compositionality and Recognitioncited by 2
  34. 2024MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representationscited by 2
  35. 2024NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?cited by 2
  36. 2025Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Bootstrappingcited by 2
  37. 2025Towards Ambiguity-Free Spatial Foundation Model: Rethinking and Decoupling Depth Ambiguitycited by 2
  38. 2025LoRA-Loop: Closing the Synthetic Replay Cycle for Continual VLM Learningcited by 2

50 most cited of 107 All 107 (JSON)