mlsanityData to 14 Oct 2026

Dataset Pruning: Reducing Training Data by Examining Generalization Influence

arXiv 2205.09329 ↗First posted 19 May 2022cs.LGcited by 133

Not on the map. Why?

Abstract (arXiv)

The great success of deep learning heavily relies on increasingly larger training data, which comes at a price of huge computational and infrastructural costs. This poses crucial questions that, do all training data contribute to model's performance? How much does each individual training sample or a sub-training-set affect the model's generalization, and how to construct the smallest subset from the entire training data as a proxy training set without significantly sacrificing the model's performance? To answer these, we propose dataset pruning, an optimization-based sample selection method that can (1) examine the influence of removing a particular set of training samples on model's generalization ability with theoretical guarantee, and (2) construct the smallest subset of training data that yields strictly constrained generalization gap. The empirically observed generalization gap of dataset pruning is substantially consistent with our theoretical expectations. Furthermore, the proposed method prunes 40% training examples on the CIFAR-10 dataset, halves the convergence time with only 1.3% test accuracy decrease, which is superior to previous score-based sample selection methods.

Key facts. Cited by 133 papers in the map (data through 14 Oct 2026). Builds on 43 of its 72 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

43 of 72 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2017Squeeze-and-Excitation Networkscited by 4,789
  6. 2014Going Deeper with Convolutionscited by 4,241
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  9. 2016Matching Networks for One Shot Learningcited by 2,255
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  12. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
31 more
  1. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  2. 2019A Closer Look at Few-shot Classificationcited by 684
  3. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  4. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  5. 2018Rethinking the Value of Network Pruningcited by 599
  6. 2018End-to-End Incremental Learningcited by 597
  7. 2020What is the State of Neural Network Pruning?cited by 583
  8. 2019Gradient based sample selection for online continual learningcited by 571
  9. 2019Importance Estimation for Neural Network Pruningcited by 541
  10. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  11. 2020Dataset Condensation with Gradient Matchingcited by 516
  12. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  13. 2021Dataset Condensation with Distribution Matchingcited by 365
  14. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  15. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  16. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  17. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 192
  18. 2021Free Lunch for Few-shot Learning: Distribution Calibrationcited by 165
  19. 2021Graph Condensation for Graph Neural Networkscited by 150
  20. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 147
  21. 2019Soft-Label Dataset Distillation and Text Dataset Distillationcited by 107
  22. 2021PartImageNet: A Large, High-Quality Dataset of Partscited by 107
  23. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 104
  24. 2018Turning Big data into tiny data: Constant-size coresets for k-means, PCA and projective clusteringcited by 94
  25. 2020Flexible Dataset Distillation: Learn Labels Instead of Imagescited by 91
  26. 2020ScaIL: Classifier Weights Scaling for Class Incremental Learningcited by 47
  27. 2016Second-Order Stochastic Optimization for Machine Learning in Linear Timecited by 14
  28. 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 12
  29. 2020Adaptive Semantic-Visual Tree for Hierarchical Embeddingscited by 8
  30. 2022Reliable Label Correction is a Good Booster When Learning with Extremely Noisy Labelscited by 4
  31. 2020Single-View 3D Object Reconstruction from Shape Priors in Memorycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 23; 2024: 45; 2025: 37; 2026: 25.

133 papers.

Sort:

By area:

  1. 2023Model Sparsity Can Simplify Machine Unlearningcited by 214
  2. 2022Training Data Influence Analysis and Estimation: A Surveycited by 135
  3. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  4. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  5. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  6. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 69
  7. 2023Data Pruning via Moving-one-Sample-outcited by 67
  8. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
  9. 2024A Survey on Deep Active Learning: Recent Advances and New Frontierscited by 54
  10. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  11. 2023Does Graph Distillation See Like Vision Dataset Counterpart?cited by 45
  12. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearningcited by 44
38 more
  1. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  2. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 32
  3. 2024ICONS: Influence Consensus for Vision-Language Data Selectioncited by 29
  4. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 27
  5. 2024Dataset Distillation by Automatic Training Trajectoriescited by 26
  6. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 26
  7. 2024A CLIP-Powered Framework for Robust and Generalizable Data Selectioncited by 21
  8. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 20
  9. 2023You Only Condense Once: Two Rules for Pruning Condensed Datasetscited by 20
  10. 2024Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Modelscited by 17
  11. 2024DANCE: Dual-View Distribution Alignment for Dataset Condensationcited by 15
  12. 2025Data Pruning by Information Maximizationcited by 15
  13. 2023GIO: Gradient Information Optimization for Training Dataset Selectioncited by 13
  14. 2023Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraintscited by 13
  15. 2025R.I.P.: Better Models by Survival of the Fittest Promptscited by 13
  16. 2025Group-Level Data Selection for Efficient Pretrainingcited by 13
  17. 2023Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learningcited by 11
  18. 2024Distilling the Knowledge in Data Pruningcited by 11
  19. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  20. 2024Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Modelscited by 10
  21. 2024Zero-Shot Coreset Selection via Iterative Subspace Samplingcited by 10
  22. 2024In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Modelscited by 9
  23. 2024Beyond Efficiency: Molecular Data Pruning for Enhanced Generalizationcited by 9
  24. 2023Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillationcited by 8
  25. 2024GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruningcited by 8
  26. 2023Deeper Understanding of Black-box Predictions via Generalized Influence Functionscited by 7
  27. 2024Evolution-aware VAriance (EVA) Coreset Selection for Medical Image Classificationcited by 7
  28. 2025Training-Free Dataset Pruning for Instance Segmentationcited by 7
  29. 2025Scale Efficient Training for Large Datasetscited by 7
  30. 2025Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspectivecited by 7
  31. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 7
  32. 2023Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Modelscited by 6
  33. 2023DavIR: Data Selection via Implicit Reward for Large Language Modelscited by 6
  34. 2023DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selectioncited by 6
  35. 20241.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Datacited by 6
  36. 2024SCAN: Bootstrapping Contrastive Pre-training for Data Efficiencycited by 6
  37. 2025Unifying Dataset Pruning and Distillation for Efficient Large-scale Compressioncited by 6
  38. 2025When Dynamic Data Selection Meets Data Augmentationcited by 6

50 most cited of 133 All 133 (JSON)