mlsanityData to 7 Oct 2026

Data pruning and neural scaling laws: fundamental limitations of score-based algorithms

arXiv 2302.06960 ↗First posted 14 Feb 2023stat.ML cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Data pruning algorithms are commonly used to reduce the memory and computational cost of the optimization process. Recent empirical results reveal that random data pruning remains a strong baseline and outperforms most existing data pruning methods in the high compression regime, i.e., where a fraction of 30% or less of the data is kept. This regime has recently attracted a lot of interest as a result of the role of data pruning in improving the so-called neural scaling laws; in [Sorscher et al.], the authors showed the need for high-quality data pruning algorithms in order to beat the sample power law. In this work, we focus on score-based data pruning algorithms and show theoretically and empirically why such algorithms fail in the high compression regime. We demonstrate ``No Free Lunch" theorems for data pruning and present calibration protocols that enhance the performance of existing pruning algorithms in this high compression regime using randomization.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

28 of 46 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2021Scaling Vision Transformerscited by 993
  4. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  5. 2019Gradient based sample selection for online continual learningcited by 568
  6. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  7. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  8. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  9. 2020Dataset Condensation with Gradient Matchingcited by 510
  10. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  11. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  12. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
16 more
  1. 2021Scaling Laws for Transfercited by 265
  2. 2018Dataset Distillationcited by 224
  3. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  4. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  5. 2019Universal Approximation with Deep Narrow Networkscited by 198
  6. 2018Universality of Deep Convolutional Neural Networkscited by 176
  7. 2018Adversarial Active Learning for Deep Networks: a Margin Based Approachcited by 155
  8. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  9. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 146
  10. 2020Contextual Diversity for Active Learningcited by 139
  11. 2021SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarioscited by 83
  12. 2016Coresets for Scalable Bayesian Logistic Regressioncited by 79
  13. 2021RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learningcited by 72
  14. 2018On Coresets for Logistic Regressioncited by 65
  15. 2017Automated Scalable Bayesian Inference via Hilbert Coresetscited by 55
  16. 2019Small-GAN: Speeding Up GAN Training Using Core-setscited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 1; 2026: 4.

11 papers.

  1. 2024How to Train Data-Efficient LLMscited by 91
  2. 2023Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learningcited by 26
  3. 2024Distilling the Knowledge in Data Pruningcited by 11
  4. 2024All models are wrong, some are useful: Model Selection with Limited Labelscited by 8
  5. 2024DRoP: Distributionally Robust Data Pruningcited by 7
  6. 2026OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Frameworkcited by 2
  7. 2024Efficient Alignment of Large Language Models via Data Samplingcited by 1
  8. 2025Dictionary-Learning-Based Data Pruning for System Identificationcited by 0
  9. 2026RCAP: Robust, Class-Aware, Probabilistic Dynamic Dataset Pruningcited by 0
  10. 2026Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillationcited by 0
  11. 2026ERank in Latent Space as an Image-Complexity and Richness Measurecited by 0