mlsanityData to 7 Oct 2026

A Closer Look at Memorization in Deep Networks

arXiv 1706.05394 ↗First posted 16 Jun 2017stat.ML cs.LGcited by 936

Not on the map. Why?

Cited by 25 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.

Key facts. Cited by 936 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 172; 2022: 204; 2023: 191; 2024: 147; 2025: 115; 2026: 106.

936 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  4. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  5. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  6. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  7. 2023Towards Unbounded Machine Unlearningcited by 261
  8. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  9. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  10. 2021Trustworthy AI: From Principles to Practicescited by 206
  11. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 182
  12. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
38 more
  1. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  2. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  3. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  4. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  5. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  6. 2021Assessing Generalization of SGD via Disagreementcited by 116
  7. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  8. 2021Understanding and Improving Early Stopping for Learning with Noisy Labelscited by 112
  9. 2021Source-free Domain Adaptation via Avatar Prototype Generation and Adaptationcited by 111
  10. 2021Regularizing Generative Adversarial Networks under Limited Datacited by 105
  11. 2023A Comprehensive Survey on Source-free Domain Adaptationcited by 104
  12. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
  13. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  14. 2021Provably End-to-end Label-Noise Learning without Anchor Pointscited by 97
  15. 2021Influence-Balanced Loss for Imbalanced Visual Classificationcited by 91
  16. 2021Augmentation Strategies for Learning with Noisy Labelscited by 86
  17. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 86
  18. 2021FINE Samples for Learning with Noisy Labelscited by 82
  19. 2021Jo-SRC: A Contrastive Approach for Combating Noisy Labelscited by 80
  20. 2021Sample Selection with Uncertainty of Losses for Learning with Noisy Labelscited by 79
  21. 2022Source-Free Unsupervised Domain Adaptation: A Surveycited by 79
  22. 2022UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learningcited by 77
  23. 2022Do Language Models Plagiarize?cited by 72
  24. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  25. 2021Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labelscited by 69
  26. 2022Agree to Disagree: Diversity through Disagreement for Better Transferabilitycited by 69
  27. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  28. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 68
  29. 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
  30. 2022Learn From All: Erasing Attention Consistency for Noisy Label Facial Expression Recognitioncited by 67
  31. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  32. 2022Synthetic Data -- what, why and how?cited by 66
  33. 2022MaskTune: Mitigating Spurious Correlations by Forcing to Explorecited by 66
  34. 2023Can Neural Network Memorization Be Localized?cited by 66
  35. 2021Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularizationcited by 62
  36. 2021Adaptive Early-Learning Correction for Segmentation from Noisy Annotationscited by 61
  37. 2021On Memorization in Probabilistic Deep Generative Modelscited by 60
  38. 2021Open-set Label Noise Can Improve Robustness Against Inherent Label Noisecited by 58

50 most cited of 936 All 936 (JSON)