# Improving the Scaling Laws of Synthetic Data with Deliberate Practice

arXiv: https://arxiv.org/abs/2502.15588 · HTML: https://mlsanity.com/paper/2502.15588/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 39 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Builds on (most cited first, 30 of 33)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners](https://mlsanity.com/paper/2306.00984/) (2023)
- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [Scaling Laws of Synthetic Images for Model Training ... for Now](https://mlsanity.com/paper/2312.04567/) (2023)
- [Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation](https://mlsanity.com/paper/2305.16289/) (2023)
- [Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation](https://mlsanity.com/paper/2402.10210/) (2024)
- [Learning Vision from Models Rivals Learning Vision from Data](https://mlsanity.com/paper/2312.17742/) (2023)
- [Strong Model Collapse](https://mlsanity.com/paper/2410.04840/) (2024)
- [Expanding Small-Scale Datasets with Guided Imagination](https://mlsanity.com/paper/2211.13976/) (2022)
- [Real-Fake: Effective Training Data Synthesis Through Distribution Matching](https://mlsanity.com/paper/2310.10402/) (2023)
- [Consistency-diversity-realism Pareto fronts of conditional image generative models](https://mlsanity.com/paper/2406.10429/) (2024)
- [Fill-Up: Balancing Long-Tailed Data with Generative Models](https://mlsanity.com/paper/2306.07200/) (2023)
- [Hessian Eigenspectra of More Realistic Nonlinear Models](https://mlsanity.com/paper/2103.01519/) (2021)
- [Feedback-guided Data Synthesis for Imbalanced Classification](https://mlsanity.com/paper/2310.00158/) (2023)
- [Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding](https://mlsanity.com/paper/2312.05328/) (2023)
- [Towards a statistical theory of data selection under weak supervision](https://mlsanity.com/paper/2309.14563/) (2023)
- [Active Prompt Learning in Vision Language Models](https://mlsanity.com/paper/2311.11178/) (2023)

## Built on by (most cited first, 17 of 17)

- [Inference-time Physics Alignment of Video Generative Models with Latent World Models](https://mlsanity.com/paper/2601.10553/) (2026)
- [A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective](https://mlsanity.com/paper/2509.16499/) (2025)
- [Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability](https://mlsanity.com/paper/2601.18778/) (2026)
- [Why Less is More (Sometimes): A Theory of Data Curation](https://mlsanity.com/paper/2511.03492/) (2025)
- [Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation](https://mlsanity.com/paper/2605.21125/) (2026)
- [Increasing the Utility of Synthetic Images through Chamfer Guidance](https://mlsanity.com/paper/2508.10631/) (2025)
- [The Intricate Dance of Prompt Complexity, Quality, Diversity, and Consistency in T2I Models](https://mlsanity.com/paper/2510.19557/) (2025)
- [A Sustainable AI Economy Needs Data Deals That Work for Generators](https://mlsanity.com/paper/2601.09966/) (2026)
- [Learnability-Guided Diffusion for Dataset Distillation](https://mlsanity.com/paper/2604.00519/) (2026)
- [Bridging Compute- and Data-Optimal Pretraining](https://mlsanity.com/paper/2607.25271/) (2026)
- [Boosting Statistic Learning with Synthetic Data from Pretrained Large Models](https://mlsanity.com/paper/2505.04992/) (2025)
- [The Anti-Ouroboros Effect: Emergent Resilience in Large Language Models from Recursive Selective Feedback](https://mlsanity.com/paper/2509.10509/) (2025)
- [Towards Application Aligned Synthetic Surgical Image Synthesis](https://mlsanity.com/paper/2509.18796/) (2025)
- [Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained Classification](https://mlsanity.com/paper/2510.24078/) (2025)
- [LiBaGS: Lightweight Boundary Gap Synthesis for Targeted Synthetic Data Selection](https://mlsanity.com/paper/2605.11231/) (2026)
- [Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics](https://mlsanity.com/paper/2607.09250/) (2026)
- [Learning-State-Aware Dynamic Generative Data Augmentation on Small-Scale Datasets](https://mlsanity.com/paper/2608.18907/) (2026)

## Abstract (arXiv)

Inspired by the principle of deliberate practice in human learning, we propose Deliberate Practice for Synthetic Data Generation (DP), a novel framework that improves sample efficiency through dynamic synthetic data generation. Prior work has shown that scaling synthetic data is inherently challenging, as naively adding new data leads to diminishing returns. To address this, pruning has been identified as a key mechanism for improving scaling, enabling models to focus on the most informative synthetic samples. Rather than generating a large dataset and pruning it afterward, DP efficiently approximates the direct generation of informative samples. We theoretically show how training on challenging, informative examples improves scaling laws and empirically validate that DP achieves better scaling performance with significantly fewer training samples and iterations. On ImageNet-100, DP generates 3.4x fewer samples and requires six times fewer iterations, while on ImageNet-1k, it generates 8x fewer samples with a 30 percent reduction in iterations, all while achieving superior performance compared to prior work.