# Accelerating Deep Learning with Dynamic Data Pruning

arXiv: https://arxiv.org/abs/2111.12621 · HTML: https://mlsanity.com/paper/2111.12621/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 24 Nov 2021.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Estimating Example Difficulty Using Variance of Gradients](https://mlsanity.com/paper/2008.11600/) (2020)
- [Curriculum Adversarial Training](https://mlsanity.com/paper/1805.04807/) (2018)

## Built on by (most cited first, 30 of 43)

- [MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models](https://mlsanity.com/paper/2406.06046/) (2024)
- [InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning](https://mlsanity.com/paper/2303.04947/) (2023)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [A CLIP-Powered Framework for Robust and Generalizable Data Selection](https://mlsanity.com/paper/2410.11215/) (2024)
- [NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Tasks](https://mlsanity.com/paper/2306.03208/) (2023)
- [Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning](https://mlsanity.com/paper/2405.19462/) (2024)
- [Beyond Efficiency: Molecular Data Pruning for Enhanced Generalization](https://mlsanity.com/paper/2409.01081/) (2024)
- [GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning](https://mlsanity.com/paper/2410.13761/) (2024)
- [Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning](https://mlsanity.com/paper/2509.23873/) (2025)
- [Scale Efficient Training for Large Datasets](https://mlsanity.com/paper/2503.13385/) (2025)
- [Language Model-Driven Data Pruning Enables Efficient Active Learning](https://mlsanity.com/paper/2410.04275/) (2024)
- [When Dynamic Data Selection Meets Data Augmentation](https://mlsanity.com/paper/2505.03809/) (2025)
- [Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization](https://mlsanity.com/paper/2406.08001/) (2024)
- [Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2410.10166/) (2024)
- [Dataset Growth](https://mlsanity.com/paper/2405.18347/) (2024)
- [Dynamic Data Pruning for Automatic Speech Recognition](https://mlsanity.com/paper/2406.18373/) (2024)
- [RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessment](https://mlsanity.com/paper/2506.21037/) (2025)
- [Partial Forward Blocking: A Novel Data Pruning Paradigm for Lossless Training Acceleration](https://mlsanity.com/paper/2506.23674/) (2025)
- [Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning](https://mlsanity.com/paper/2507.12998/) (2025)
- [Active Semi-Supervised Learning by Exploring Per-Sample Uncertainty and Consistency](https://mlsanity.com/paper/2303.08978/) (2023)
- [Rethinking Representativeness and Diversity in Dynamic Data Selection](https://mlsanity.com/paper/2603.04981/) (2026)
- [Data Agent: Learning to Select Data via End-to-End Dynamic Optimization](https://mlsanity.com/paper/2603.07433/) (2026)
- [OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Framework](https://mlsanity.com/paper/2606.08574/) (2026)
- [Representative Subset Selection for Efficient Fine-Tuning in Self-Supervised Speech Recognition](https://mlsanity.com/paper/2203.09829/) (2022)
- [Identify ambiguous tasks combining crowdsourced labels by weighting Areas Under the Margin](https://mlsanity.com/paper/2209.15380/) (2022)
- [Weight Compander: A Simple Weight Reparameterization for Regularization](https://mlsanity.com/paper/2306.16993/) (2023)
- [TED: Accelerate Model Training by Internal Generalization](https://mlsanity.com/paper/2405.03228/) (2024)
- [Robust Time Series Forecasting with Non-Heavy-Tailed Gaussian Loss-Weighted Sampler](https://mlsanity.com/paper/2406.13871/) (2024)
- [Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning](https://mlsanity.com/paper/2507.12750/) (2025)
- [EMP: Enhance Memory in Data Pruning](https://mlsanity.com/paper/2408.16031/) (2024)

## Abstract (arXiv)

Deep learning's success has been attributed to the training of large, overparameterized models on massive amounts of data. As this trend continues, model training has become prohibitively costly, requiring access to powerful computing systems to train state-of-the-art networks. A large body of research has been devoted to addressing the cost per iteration of training through various model compression techniques like pruning and quantization. Less effort has been spent targeting the number of iterations. Previous work, such as forget scores and GraNd/EL2N scores, address this problem by identifying important samples within a full dataset and pruning the remaining samples, thereby reducing the iterations per epoch. Though these methods decrease the training time, they use expensive static scoring algorithms prior to training. When accounting for the scoring mechanism, the total run time is often increased. In this work, we address this shortcoming with dynamic data pruning algorithms. Surprisingly, we find that uniform random dynamic pruning can outperform the prior work at aggressive pruning rates. We attribute this to the existence of "sometimes" samples -- points that are important to the learned decision boundary only some of the training time. To better exploit the subtlety of sometimes samples, we propose two algorithms, based on reinforcement learning techniques, to dynamically prune samples and achieve even higher accuracy than the random dynamic method. We test all our methods against a full-dataset baseline and the prior work on CIFAR-10 and CIFAR-100, and we can reduce the training time by up to 2x without significant performance loss. Our results suggest that data pruning should be understood as a dynamic process that is closely tied to a model's training trajectory, instead of a static step based solely on the dataset alone.