mlsanityData to 7 Oct 2026

The Pitfalls of Memorization: When Memorization Hurts Generalization

arXiv 2412.07684 ↗First posted 10 Dec 2024cs.LG cs.AI stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these explanations.This behavior leads to poor generalization when the learned explanations rely on spurious correlations. In this work, we formalize the interplay between memorization and generalization, showing that spurious correlations would particularly lead to poor generalization when are combined with memorization. Memorization can reduce training loss to zero, leaving no incentive to learn robust, generalizable patterns. To address this, we propose memorization-aware training (MAT), which uses held-out predictions as a signal of memorization to shift a model's logits. MAT encourages learning robust patterns invariant across distributions, improving generalization under distribution shifts.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 62 of its 74 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

62 of 74 references.

References · 62

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  6. 2019Invariant Risk Minimizationcited by 1,832
  7. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  8. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  9. 2017A Closer Look at Memorization in Deep Networkscited by 936
  10. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  11. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
38 more
  1. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2020Long-tail learning via logit adjustmentcited by 605
  5. 2019Benign Overfitting in Linear Regressioncited by 544
  6. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  7. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  8. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  9. 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 397
  10. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  11. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  12. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  13. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  14. 2020Environment Inference for Invariant Learningcited by 306
  15. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  16. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  17. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  18. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  19. 2021Simple data balancing achieves competitive worst-group-accuracycited by 177
  20. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  21. 2020Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scanscited by 158
  22. 2019REPAIR: Removing Representation Bias by Dataset Resamplingcited by 157
  23. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  24. 2019Harmless interpolation of noisy data in regressioncited by 121
  25. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  26. 2018Does data interpolation contradict statistical optimality?cited by 95
  27. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  28. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  29. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  30. 2023Simple and Fast Group Robustness by Automatic Feature Reweightingcited by 68
  31. 2023Can Neural Network Memorization Be Localized?cited by 66
  32. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  33. 2019Adjusting Decision Boundary for Class Imbalanced Learningcited by 52
  34. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  35. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  36. 2022UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixupcited by 34
  37. 2019Identity Crisis: Memorization and Generalization under Extreme Overparameterizationcited by 33
  38. 2023Feedback-guided Data Synthesis for Imbalanced Classificationcited by 33

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 8.

16 papers.

Sort:

By area:

  1. 2024Titans: Learning to Memorize at Test Timecited by 303
  2. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  3. 2025Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledgecited by 9
  4. 2025Memorize or Generalize? Evaluating LLM Code Generation with Code Rewritingcited by 6
  5. 2025Rote Learning Considered Useful: Generalizing over Memorized Data in LLMscited by 5
  6. 2025Fodor and Pylyshyn's Legacy: Still No Human-like Systematic Compositionality in Neural Networkscited by 3
  7. 2025Generalization or Memorization: Dynamic Decoding for Mode Steeringcited by 3
  8. 2026GSS: Gated Subspace Steering for Selective Memorization Mitigation in LLMscited by 2
  9. 2026Proteus: Incremental Memory Activation for Long-Context Sequence Modelingcited by 1
  10. 2025Memorization in Graph Neural Networkscited by 0
  11. 2026Anatomy of a failure: When, how, and why deep vision fails in scientific domainscited by 0
  12. 2026Cumulative Meta-Learning from Active Learning Queries for Robustness to Spurious Correlationscited by 0
4 more
  1. 2026Mitigating Spurious Correlations with Memorization-Guided Dataset De-Biasingcited by 0
  2. 2026Random Label Prediction Heads for Studying Memorization in Deep Neural Networkscited by 0
  3. 2026What do Reward Models Memorize?cited by 0
  4. 2026Whitening Improves Robustness to Spurious Correlations in Linear Probescited by 0