mlsanityData to 7 Oct 2026

Beyond neural scaling laws: beating power law scaling via data pruning

arXiv 2206.14486 ↗First posted 29 Jun 2022cs.LG cs.AI cs.CVcited by 519

Not on the map. Why?

Abstract (arXiv)

Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how in theory we can break beyond power law scaling and potentially even reduce it to exponential scaling instead if we have access to a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this improved scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling in practice on ResNets trained on CIFAR-10, SVHN, and ImageNet. Next, given the importance of finding high-quality pruning metrics, we perform the first large-scale benchmarking study of ten different data pruning metrics on ImageNet. We find most existing high performing metrics scale poorly to ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metrics. Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.

Key facts. Cited by 519 papers in the map (data through 7 Oct 2026). Builds on 37 of its 70 references that are in the map. First posted on arXiv 29 Jun 2022.

Builds on

37 of 70 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  11. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  12. 2021Scaling Vision Transformerscited by 993
25 more
  1. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  2. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  3. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  4. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  5. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  6. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  7. 2021Explaining Neural Scaling Lawscited by 353
  8. 2021Scaling Laws for Transfercited by 265
  9. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  10. 2018Generalisation in humans and deep neural networkscited by 237
  11. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  12. 2021Partial success in closing the gap between human and machine visioncited by 185
  13. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  14. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  15. 2020Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistencycited by 77
  16. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  17. 2021PASS: An ImageNet replacement for self-supervised pretraining without humanscited by 51
  18. 2019Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Needcited by 43
  19. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  20. 2021Active Learning at the ImageNet Scalecited by 28
  21. 2021Scaling Laws for Deep Learningcited by 25
  22. 2013Statistical mechanics of complex neural systems and high dimensional datacited by 15
  23. 2021Estimating informativeness of samples with Smooth Unique Informationcited by 15
  24. 2019Large deviations for the perceptron model and consequences for active learningcited by 7
  25. 2019Active online learning in the binary perceptron problemcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 21; 2023: 113; 2024: 139; 2025: 132; 2026: 114.

519 papers.

Sort:

By area:

  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  2. 2023Large Language Modelscited by 774
  3. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  4. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  5. 2023Scaling Data-Constrained Language Modelscited by 391
  6. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  7. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  8. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  9. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  10. 2023Demystifying CLIP Datacited by 265
  11. 2023Challenges and Applications of Large Language Modelscited by 251
  12. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
38 more
  1. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  2. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  3. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  4. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  5. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  6. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 115
  7. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  8. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  9. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  10. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  11. 2022Broken Neural Scaling Lawscited by 102
  12. 2022A Solvable Model of Neural Scaling Lawscited by 101
  13. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  14. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  15. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 93
  16. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  17. 2024How to Train Data-Efficient LLMscited by 91
  18. 2023Data Distillation: A Surveycited by 86
  19. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  20. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  21. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  22. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  23. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  24. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  25. 2023Data Pruning via Moving-one-Sample-outcited by 67
  26. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
  27. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  28. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  29. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  30. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  31. 2024Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approachcited by 48
  32. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  33. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
  34. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  35. 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 43
  36. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  37. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 43
  38. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41

50 most cited of 519 All 519 (JSON)