mlsanityData to 7 Oct 2026

Improving the Scaling Laws of Synthetic Data with Deliberate Practice

arXiv 2502.15588 ↗First posted 21 Feb 2025cs.LG cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Inspired by the principle of deliberate practice in human learning, we propose Deliberate Practice for Synthetic Data Generation (DP), a novel framework that improves sample efficiency through dynamic synthetic data generation. Prior work has shown that scaling synthetic data is inherently challenging, as naively adding new data leads to diminishing returns. To address this, pruning has been identified as a key mechanism for improving scaling, enabling models to focus on the most informative synthetic samples. Rather than generating a large dataset and pruning it afterward, DP efficiently approximates the direct generation of informative samples. We theoretically show how training on challenging, informative examples improves scaling laws and empirically validate that DP achieves better scaling performance with significantly fewer training samples and iterations. On ImageNet-100, DP generates 3.4x fewer samples and requires six times fewer iterations, while on ImageNet-1k, it generates 8x fewer samples with a 30 percent reduction in iterations, all while achieving superior performance compared to prior work.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 33 of its 39 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

33 of 39 references.

References · 33

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2019Natural Adversarial Examplescited by 1,401
  10. 2019Contrastive Multiview Codingcited by 1,297
  11. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
21 more
  1. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  2. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  3. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 171
  4. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  5. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  6. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  7. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  8. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  9. 2024Strong Model Collapsecited by 62
  10. 2022Expanding Small-Scale Datasets with Guided Imaginationcited by 61
  11. 2023Real-Fake: Effective Training Data Synthesis Through Distribution Matchingcited by 35
  12. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  13. 2023Fill-Up: Balancing Long-Tailed Data with Generative Modelscited by 34
  14. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  15. 2023Feedback-guided Data Synthesis for Imbalanced Classificationcited by 33
  16. 2023Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understandingcited by 28
  17. 2023Towards a statistical theory of data selection under weak supervisioncited by 25
  18. 2023Active Prompt Learning in Vision Language Modelscited by 25
  19. 2024GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learningcited by 8
  20. 2023Instance-Conditioned GAN Data Augmentation for Representation Learningcited by 7
  21. 2024Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theorycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 9.

17 papers.

Sort:

By area:

  1. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 32
  2. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  3. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  4. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  5. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  6. 2025Increasing the Utility of Synthetic Images through Chamfer Guidancecited by 6
  7. 2025The Intricate Dance of Prompt Complexity, Quality, Diversity, and Consistency in T2I Modelscited by 6
  8. 2026A Sustainable AI Economy Needs Data Deals That Work for Generatorscited by 3
  9. 2026Learnability-Guided Diffusion for Dataset Distillationcited by 2
  10. 2026Bridging Compute- and Data-Optimal Pretrainingcited by 1
  11. 2025Boosting Statistic Learning with Synthetic Data from Pretrained Large Modelscited by 0
  12. 2025The Anti-Ouroboros Effect: Emergent Resilience in Large Language Models from Recursive Selective Feedbackcited by 0
5 more
  1. 2025Towards Application Aligned Synthetic Surgical Image Synthesiscited by 0
  2. 2025Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained Classificationcited by 0
  3. 2026LiBaGS: Lightweight Boundary Gap Synthesis for Targeted Synthetic Data Selectioncited by 0
  4. 2026Influence Diagnostics in High-dimensional M-estimation: Precise Asymptoticscited by 0
  5. 2026Learning-State-Aware Dynamic Generative Data Augmentation on Small-Scale Datasetscited by 0