# Large-scale Dataset Pruning with Dynamic Uncertainty

arXiv: https://arxiv.org/abs/2306.05175 · HTML: https://mlsanity.com/paper/2306.05175/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 50 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning](https://mlsanity.com/paper/2012.10630/) (2020)
- [Coresets via Bilevel Optimization for Continual Learning and Streaming](https://mlsanity.com/paper/2006.03875/) (2020)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [Adversarial Active Learning for Deep Networks: a Margin Based Approach](https://mlsanity.com/paper/1802.09841/) (2018)
- [Active Learning by Acquiring Contrastive Examples](https://mlsanity.com/paper/2109.03764/) (2021)
- [Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples](https://mlsanity.com/paper/1704.07433/) (2017)

## Built on by (most cited first, 30 of 63)

- [Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning](https://mlsanity.com/paper/2311.13613/) (2023)
- [Effective pruning of web-scale datasets based on complexity of concept clusters](https://mlsanity.com/paper/2401.04578/) (2024)
- [IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models](https://mlsanity.com/paper/2310.10873/) (2023)
- [Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty](https://mlsanity.com/paper/2502.06905/) (2025)
- [Data Pruning by Information Maximization](https://mlsanity.com/paper/2506.01701/) (2025)
- [Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints](https://mlsanity.com/paper/2311.08675/) (2023)
- [Zero-Shot Coreset Selection via Iterative Subspace Sampling](https://mlsanity.com/paper/2411.15349/) (2024)
- [DRoP: Distributionally Robust Data Pruning](https://mlsanity.com/paper/2404.05579/) (2024)
- [Scale Efficient Training for Large Datasets](https://mlsanity.com/paper/2503.13385/) (2025)
- [Severing Spurious Correlations with Data Pruning](https://mlsanity.com/paper/2503.18258/) (2025)
- [Language Model-Driven Data Pruning Enables Efficient Active Learning](https://mlsanity.com/paper/2410.04275/) (2024)
- [When Dynamic Data Selection Meets Data Augmentation](https://mlsanity.com/paper/2505.03809/) (2025)
- [Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2410.10166/) (2024)
- [A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design](https://mlsanity.com/paper/2607.09084/) (2026)
- [Fairness Under Demographic Scarce Regime](https://mlsanity.com/paper/2307.13081/) (2023)
- [Dynamic Data Pruning for Automatic Speech Recognition](https://mlsanity.com/paper/2406.18373/) (2024)
- [Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation](https://mlsanity.com/paper/2407.05693/) (2024)
- [The Impact of Coreset Selection on Spurious Correlations and Group Robustness](https://mlsanity.com/paper/2507.11690/) (2025)
- [SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe](https://mlsanity.com/paper/2410.05248/) (2024)
- [Data Pruning in Generative Diffusion Models](https://mlsanity.com/paper/2411.12523/) (2024)
- [Partial Forward Blocking: A Novel Data Pruning Paradigm for Lossless Training Acceleration](https://mlsanity.com/paper/2506.23674/) (2025)
- [Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning](https://mlsanity.com/paper/2507.12998/) (2025)
- [Coresets from Trajectories: Selecting Data via Correlation of Loss Differences](https://mlsanity.com/paper/2508.20230/) (2025)
- [TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuning](https://mlsanity.com/paper/2510.07118/) (2025)
- [Learnability-Guided Diffusion for Dataset Distillation](https://mlsanity.com/paper/2604.00519/) (2026)
- [Efficient Architecture Search via Bi-level Data Pruning](https://mlsanity.com/paper/2312.14200/) (2023)
- [Large-Scale Dataset Pruning in Adversarial Training through Data Importance Extrapolation](https://mlsanity.com/paper/2406.13283/) (2024)
- [Bridging Distribution Shift and AI Safety: Conceptual and Methodological Synergies](https://mlsanity.com/paper/2505.22829/) (2025)
- [Effective Data Pruning through Score Extrapolation](https://mlsanity.com/paper/2506.09010/) (2025)
- [Data Agent: Learning to Select Data via End-to-End Dynamic Optimization](https://mlsanity.com/paper/2603.07433/) (2026)

## Abstract (arXiv)

The state of the art of many learning tasks, e.g., image classification, is advanced by collecting larger datasets and then training larger models on them. As the outcome, the increasing computational cost is becoming unaffordable. In this paper, we investigate how to prune the large-scale datasets, and thus produce an informative subset for training sophisticated deep models with negligible performance drop. We propose a simple yet effective dataset pruning method by exploring both the prediction uncertainty and training dynamics. We study dataset pruning by measuring the variation of predictions during the whole training process on large-scale datasets, i.e., ImageNet-1K and ImageNet-21K, and advanced models, i.e., Swin Transformer and ConvNeXt. Extensive experimental results indicate that our method outperforms the state of the art and achieves 25% lossless pruning ratio on both ImageNet-1K and ImageNet-21K. The code and pruned datasets are available at https://github.com/BAAI-DCAI/Dataset-Pruning.