mlsanityData to 14 Oct 2026

Deep Learning on a Data Diet: Finding Important Examples Early in Training

arXiv 2107.07075 ↗First posted 15 Jul 2021cs.LGcited by 533

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent success in deep learning has partially been driven by training increasingly overparametrized networks on ever larger datasets. It is therefore natural to ask: how much of the data is superfluous, which examples are important for generalization, and how do we find them? In this work, we make the striking observation that, in standard vision datasets, simple scores averaged over several weight initializations can be used to identify important examples very early in training. We propose two such scores -- the Gradient Normed (GraNd) and the Error L2-Norm (EL2N) scores -- and demonstrate their efficacy on a range of architectures and datasets by pruning significant fractions of training data without sacrificing test accuracy. In fact, using EL2N scores calculated a few epochs into training, we can prune half of the CIFAR10 training set while slightly improving test accuracy. Furthermore, for a given dataset, EL2N scores from one architecture or hyperparameter configuration generalize to other configurations. Compared to recent work that prunes data by discarding examples that are rarely forgotten over the course of training, our scores use only local information early in training. We also use our scores to detect noisy examples and study training dynamics through the lens of important examples -- we investigate how the data distribution shapes the loss surface and identify subspaces of the model's data representation that are relatively stable over training.

Key facts. Cited by 533 papers in the map (data through 14 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 15 Jul 2021.

Builds on

25 of 39 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  6. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  7. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  8. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  9. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  10. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  11. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  12. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
13 more
  1. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 226
  2. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  3. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  4. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  5. 2017Deep Active Learning for Named Entity Recognitioncited by 108
  6. 2020Estimating Example Difficulty Using Variance of Gradientscited by 97
  7. 2018Turning Big data into tiny data: Constant-size coresets for k-means, PCA and projective clusteringcited by 94
  8. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  9. 2016Coresets for Scalable Bayesian Logistic Regressioncited by 79
  10. 2018Bayesian Coreset Construction via Greedy Iterative Geodesic Ascentcited by 68
  11. 2018On Coresets for Logistic Regressioncited by 67
  12. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  13. 2020Coresets for Robust Training of Neural Networks against Noisy Labelscited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 39; 2023: 92; 2024: 142; 2025: 132; 2026: 122.

533 papers.

Sort:

By area:

  1. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  2. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  3. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  4. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  5. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  6. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  7. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  8. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  9. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  10. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  11. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 147
  12. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 133
38 more
  1. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  2. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
  3. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 109
  4. 2022Dataset Condensation with Contrastive Signalscited by 105
  5. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  6. 2023Finding Support Examples for In-Context Learningcited by 104
  7. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  8. 2024How to Train Data-Efficient LLMscited by 93
  9. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  10. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  11. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  12. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 69
  13. 2023Data Pruning via Moving-one-Sample-outcited by 67
  14. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
  15. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  16. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 61
  17. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52
  18. 2024Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approachcited by 50
  19. 2024Are we making progress in unlearning? Findings from the first NeurIPS unlearning competitioncited by 50
  20. 2023Dataset Quantizationcited by 48
  21. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 47
  22. 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
  23. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearningcited by 44
  24. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 44
  25. 2021Accelerating Deep Learning with Dynamic Data Pruningcited by 43
  26. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  27. 2022Towards A Proactive ML Approach for Detecting Backdoor Poison Samplescited by 40
  28. 2023A Survey on Efficient Training of Transformerscited by 39
  29. 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
  30. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  31. 2023Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruningcited by 38
  32. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  33. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  34. 2023Data Valuation Without Training of a Modelcited by 35
  35. 2024Effective pruning of web-scale datasets based on complexity of concept clusterscited by 34
  36. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 34
  37. 2023Loss-Curvature Matching for Dataset Selection and Condensationcited by 33
  38. 2024Towards Efficient Replay in Federated Incremental Learningcited by 33

50 most cited of 533 All 533 (JSON)