mlsanityData to 7 Oct 2026

An Empirical Study of Example Forgetting during Deep Neural Network Learning

arXiv 1812.05159 ↗First posted 12 Dec 2018cs.LG stat.MLcited by 669

Not on the map. Why?

Abstract (arXiv)

Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a `forgetting event' to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Across several benchmark data sets, we find that: (i) certain examples are forgotten with high frequency, and some not at all; (ii) a data set's (un)forgettable examples generalize across neural architectures; and (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.

Key facts. Cited by 669 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 55; 2022: 87; 2023: 137; 2024: 156; 2025: 127; 2026: 107.

669 papers.

Sort:

By area:

  1. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  2. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  3. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  6. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  7. 2021Dataset Condensation with Distribution Matchingcited by 362
  8. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  9. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  10. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  12. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
38 more
  1. 2023Demystifying CLIP Datacited by 265
  2. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  3. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  4. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  5. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  6. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 146
  7. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  8. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  9. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 137
  10. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  11. 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 124
  12. 2023Improved Distribution Matching for Dataset Condensationcited by 124
  13. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  14. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  15. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  16. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 115
  17. 2022Dataset Distillation via Factorizationcited by 115
  18. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  19. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  20. 2021Case-based Reasoning for Natural Language Queries over Knowledge Basescited by 107
  21. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  22. 2023Finding Support Examples for In-Context Learningcited by 104
  23. 2022Membership Inference Attacks by Exploiting Loss Trajectorycited by 93
  24. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 93
  25. 2024How to Train Data-Efficient LLMscited by 91
  26. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  27. 2024What makes unlearning hard and what to do about itcited by 89
  28. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  29. 2023Data Distillation: A Surveycited by 86
  30. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 83
  31. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  32. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
  33. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  34. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 74
  35. 2021Learning Fast Sample Re-weighting Without Reward Datacited by 69
  36. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  37. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  38. 2023Data Pruning via Moving-one-Sample-outcited by 67

50 most cited of 669 All 669 (JSON)