# Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty

arXiv: https://arxiv.org/abs/2502.06905 · HTML: https://mlsanity.com/paper/2502.06905/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Coverage-centric Coreset Selection for High Pruning Rates](https://mlsanity.com/paper/2210.15809/) (2022)
- [Selective-Supervised Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2203.04181/) (2022)
- [Iterative Learning with Open-set Noisy Labels](https://mlsanity.com/paper/1804.00092/) (2018)
- [Characterizing Structural Regularities of Labeled Data in Overparameterized Models](https://mlsanity.com/paper/2002.03206/) (2020)
- [InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning](https://mlsanity.com/paper/2303.04947/) (2023)
- [D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning](https://mlsanity.com/paper/2310.07931/) (2023)
- [Large-scale Dataset Pruning with Dynamic Uncertainty](https://mlsanity.com/paper/2306.05175/) (2023)
- [Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning](https://mlsanity.com/paper/2311.13613/) (2023)
- [Probabilistic Bilevel Coreset Selection](https://mlsanity.com/paper/2301.09880/) (2023)
- [BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Ranges](https://mlsanity.com/paper/2406.03057/) (2024)
- [Instance Correction for Learning with Open-set Noisy Labels](https://mlsanity.com/paper/2106.00455/) (2021)
- [Data Pruning via Separability, Integrity, and Model Uncertainty-Aware Importance Sampling](https://mlsanity.com/paper/2409.13915/) (2024)

## Built on by (most cited first, 22 of 22)

- [A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances](https://mlsanity.com/paper/2505.17799/) (2025)
- [AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners](https://mlsanity.com/paper/2505.16322/) (2025)
- [Model Parallelism With Subnetwork Data Parallelism](https://mlsanity.com/paper/2507.09029/) (2025)
- [Coresets from Trajectories: Selecting Data via Correlation of Loss Differences](https://mlsanity.com/paper/2508.20230/) (2025)
- [TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuning](https://mlsanity.com/paper/2510.07118/) (2025)
- [Bridging Distribution Shift and AI Safety: Conceptual and Methodological Synergies](https://mlsanity.com/paper/2505.22829/) (2025)
- [Data Agent: Learning to Select Data via End-to-End Dynamic Optimization](https://mlsanity.com/paper/2603.07433/) (2026)
- [Measuring Fine-Grained Relatedness in Multitask Learning via Data Attribution](https://mlsanity.com/paper/2505.21438/) (2025)
- [Investigating Data Pruning for Pretraining Biological Foundation Models at Scale](https://mlsanity.com/paper/2512.12932/) (2025)
- [TopoPrune: Robust Data Pruning via Unified Latent Space Topology](https://mlsanity.com/paper/2602.02739/) (2026)
- [GRACE: A Dynamic Coreset Selection Framework for Large Language Model Optimization](https://mlsanity.com/paper/2604.11810/) (2026)
- [Rethinking Dataset Distillation: Hard Truths about Soft Labels](https://mlsanity.com/paper/2604.18811/) (2026)
- [An Adaptive Data cleaning Framework for Noisy Label Detection](https://mlsanity.com/paper/2606.07086/) (2026)
- [Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning](https://mlsanity.com/paper/2606.25488/) (2026)
- [Extending Dataset Pruning to Object Detection: A Variance-based Approach](https://mlsanity.com/paper/2505.17245/) (2025)
- [Dataset Color Quantization: A Training-Oriented Framework for Dataset-Level Compression](https://mlsanity.com/paper/2602.20650/) (2026)
- [Beyond Loss Values: Robust Dynamic Pruning via Loss Trajectory Alignment](https://mlsanity.com/paper/2604.07306/) (2026)
- [Constraint-Data-Value-Maximization: Utilizing Data Attribution for Effective Data Pruning in Low-Data Environments](https://mlsanity.com/paper/2605.11312/) (2026)
- [Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training](https://mlsanity.com/paper/2605.14773/) (2026)
- [Label-Efficient Dataset Pruning via Semi-Supervised Pseudo-Labeling](https://mlsanity.com/paper/2605.23198/) (2026)
- [Unmasking Removal-Budget Confounding: A Matched Operating-Point Evaluation Framework for Adaptive Data Cleaning](https://mlsanity.com/paper/2608.06511/) (2026)
- [When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation](https://mlsanity.com/paper/2609.34489/) (2026)

## Abstract (arXiv)

Recent advances in deep learning rely heavily on massive datasets, leading to substantial storage and training costs. Dataset pruning aims to alleviate this demand by discarding redundant examples. However, many existing methods require training a model with a full dataset over a large number of epochs before being able to prune the dataset, which ironically makes the pruning process more expensive than just training the model on the entire dataset. To overcome this limitation, we introduce a Difficulty and Uncertainty-Aware Lightweight (DUAL) score, which aims to identify important samples from the early training stage by considering both example difficulty and prediction uncertainty. To address a catastrophic accuracy drop at an extreme pruning, we further propose a ratio-adaptive sampling using Beta distribution. Experiments on various datasets and learning scenarios such as image classification with label noise and image corruption, and model architecture generalization demonstrate the superiority of our method over previous state-of-the-art (SOTA) approaches. Specifically, on ImageNet-1k, our method reduces the time cost for pruning to 66% compared to previous methods while achieving a SOTA, specifically 60% test accuracy at a 90% pruning ratio. On CIFAR datasets, the time cost is reduced to just 15% while maintaining SOTA performance.