mlsanityData to 14 Oct 2026

Large-scale Dataset Pruning with Dynamic Uncertainty

arXiv 2306.05175 ↗First posted 8 Jun 2023cs.LG cs.CVcited by 63

Not on the map. Why?

Abstract (arXiv)

The state of the art of many learning tasks, e.g., image classification, is advanced by collecting larger datasets and then training larger models on them. As the outcome, the increasing computational cost is becoming unaffordable. In this paper, we investigate how to prune the large-scale datasets, and thus produce an informative subset for training sophisticated deep models with negligible performance drop. We propose a simple yet effective dataset pruning method by exploring both the prediction uncertainty and training dynamics. We study dataset pruning by measuring the variation of predictions during the whole training process on large-scale datasets, i.e., ImageNet-1K and ImageNet-21K, and advanced models, i.e., Swin Transformer and ConvNeXt. Extensive experimental results indicate that our method outperforms the state of the art and achieves 25% lossless pruning ratio on both ImageNet-1K and ImageNet-21K. The code and pruned datasets are available at https://github.com/BAAI-DCAI/Dataset-Pruning.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

38 of 50 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2022A ConvNet for the 2020scited by 3,579
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  7. 2019Natural Adversarial Examplescited by 1,408
  8. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  9. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  10. 2017Deep Bayesian Active Learning with Image Datacited by 819
  11. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
26 more
  1. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  2. 2021ImageNet-21K Pretraining for the Massescited by 613
  3. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  5. 2020Dataset Condensation with Gradient Matchingcited by 516
  6. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  7. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  8. 2021Dataset Condensation with Distribution Matchingcited by 365
  9. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  10. 2020Are we done with ImageNet?cited by 333
  11. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  12. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  13. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 226
  14. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 192
  15. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 181
  16. 2018Adversarial Active Learning for Deep Networks: a Margin Based Approachcited by 158
  17. 2021Active Learning by Acquiring Contrastive Examplescited by 156
  18. 2017Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samplescited by 147
  19. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 147
  20. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 133
  21. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
  22. 2017Deep Active Learning for Named Entity Recognitioncited by 108
  23. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 104
  24. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  25. 2022Synthesizing Informative Training Samples with GANcited by 77
  26. 2021RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learningcited by 74

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 12; 2025: 23; 2026: 23.

63 papers.

Sort:

By area:

  1. 2023Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruningcited by 38
  2. 2024Effective pruning of web-scale datasets based on complexity of concept clusterscited by 34
  3. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 27
  4. 2025Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertaintycited by 22
  5. 2025Data Pruning by Information Maximizationcited by 15
  6. 2023Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraintscited by 13
  7. 2024Zero-Shot Coreset Selection via Iterative Subspace Samplingcited by 10
  8. 2024DRoP: Distributionally Robust Data Pruningcited by 7
  9. 2025Scale Efficient Training for Large Datasetscited by 7
  10. 2025Severing Spurious Correlations with Data Pruningcited by 7
  11. 2024Language Model-Driven Data Pruning Enables Efficient Active Learningcited by 6
  12. 2025When Dynamic Data Selection Meets Data Augmentationcited by 6
38 more
  1. 2024Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Modelscited by 5
  2. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5
  3. 2023Fairness Under Demographic Scarce Regimecited by 4
  4. 2024Dynamic Data Pruning for Automatic Speech Recognitioncited by 4
  5. 2024Sub-SA: Strengthen In-context Learning via Submodular Selective Annotationcited by 4
  6. 2025The Impact of Coreset Selection on Spurious Correlations and Group Robustnesscited by 4
  7. 2024SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipecited by 3
  8. 2024Data Pruning in Generative Diffusion Modelscited by 3
  9. 2025Partial Forward Blocking: A Novel Data Pruning Paradigm for Lossless Training Accelerationcited by 3
  10. 2025Differential-informed Sample Selection Accelerates Multimodal Contrastive Learningcited by 3
  11. 2025Coresets from Trajectories: Selecting Data via Correlation of Loss Differencescited by 3
  12. 2025TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuningcited by 3
  13. 2026Learnability-Guided Diffusion for Dataset Distillationcited by 3
  14. 2023Efficient Architecture Search via Bi-level Data Pruningcited by 2
  15. 2024Large-Scale Dataset Pruning in Adversarial Training through Data Importance Extrapolationcited by 2
  16. 2025Bridging Distribution Shift and AI Safety: Conceptual and Methodological Synergiescited by 2
  17. 2025Effective Data Pruning through Score Extrapolationcited by 2
  18. 2026Data Agent: Learning to Select Data via End-to-End Dynamic Optimizationcited by 2
  19. 2026OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Frameworkcited by 2
  20. 2024Robust Time Series Forecasting with Non-Heavy-Tailed Gaussian Loss-Weighted Samplercited by 1
  21. 2024Are Sparse Neural Networks Better Hard Sample Learners?cited by 1
  22. 2025Finding the Muses: Identifying Coresets through Loss Trajectoriescited by 1
  23. 2025Measuring Fine-Grained Relatedness in Multitask Learning via Data Attributioncited by 1
  24. 2025Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learningcited by 1
  25. 2025Investigating Data Pruning for Pretraining Biological Foundation Models at Scalecited by 1
  26. 2026TopoPrune: Robust Data Pruning via Unified Latent Space Topologycited by 1
  27. 2026Rethinking Dataset Distillation: Hard Truths about Soft Labelscited by 1
  28. 2026Exploring and Exploiting Stability in Latent Flow Matchingcited by 1
  29. 2026Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruningcited by 1
  30. 2025A Causal Framework for Aligning Image Quality Metrics and Deep Neural Network Robustnesscited by 0
  31. 2025DiffProb: Data Pruning for Face Recognitioncited by 0
  32. 2025Extending Dataset Pruning to Object Detection: A Variance-based Approachcited by 0
  33. 2025Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selectioncited by 0
  34. 2025Efficient Training of Deep Networks using Guided Spectral Data Selection: A Step Toward Learning What You Needcited by 0
  35. 2025Surprisingly High Redundancy in Electronic Structure Data Across Materials Explained by Low Intrinsic Dimensionalitycited by 0
  36. 2025UNSEEN: Enhancing Dataset Pruning from a Generalization Perspectivecited by 0
  37. 2026Position: Stop Preaching and Start Practising Data Frugality for Responsible Development of AIcited by 0
  38. 2026SCOPE: Semantic Coreset with Orthogonal Projection Embeddings for Federated learningcited by 0

50 most cited of 63 All 63 (JSON)