mlsanityData to 14 Oct 2026

Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

arXiv 2205.10770 ↗First posted 22 May 2022cs.CLcited by 220

Not on the map. Why?

Cited by 38 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite their wide adoption, the underlying training and memorization dynamics of very large language models is not well understood. We empirically study exact memorization in causal and masked language modeling, across model sizes and throughout the training process. We measure the effects of dataset size, learning rate, and model size on memorization, finding that larger language models memorize training data faster across all settings. Surprisingly, we show that larger models can memorize a larger portion of the data before over-fitting and tend to forget less throughout the training process. We also analyze the memorization dynamics of different parts of speech and find that models memorize nouns and numbers first; we hypothesize and provide empirical evidence that nouns and numbers act as a unique identifier for memorizing individual training examples. Together, these findings present another piece of the broader puzzle of trying to understand what actually improves as models get bigger.

Key facts. Cited by 220 papers in the map (data through 14 Oct 2026). Builds on 67 of its 107 references that are in the map. First posted on arXiv 22 May 2022.

Builds on

67 of 107 references.

References · 67

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
38 more
  1. 2020Extracting Training Data from Large Language Modelscited by 2,189
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2017Mixed Precision Trainingcited by 1,156
  11. 2019Machine Unlearningcited by 1,061
  12. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  13. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  14. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  15. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  16. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  17. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  18. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  19. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  20. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  21. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  22. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  23. 2021Explaining Neural Scaling Lawscited by 363
  24. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  25. 2021Scaling Laws for Transfercited by 272
  26. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  27. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  28. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  29. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  30. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  31. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 175
  32. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  33. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  34. 2022A Review on Language Models as Knowledge Basescited by 157
  35. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  36. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  37. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  38. 2020The Early Phase of Neural Network Trainingcited by 115

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 12; 2023: 48; 2024: 63; 2025: 61; 2026: 36.

220 papers.

Sort:

By area:

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  3. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  4. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  5. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 424
  6. 2023Augmented Language Models: a Surveycited by 403
  7. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  8. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 224
  9. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  10. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  11. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  12. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
38 more
  1. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 169
  2. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  3. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  4. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  5. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  6. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  7. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  8. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 100
  9. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  10. 2023On Provable Copyright Protection for Generative Modelscited by 98
  11. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 98
  12. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  13. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  14. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 89
  15. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 83
  16. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 83
  17. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 79
  18. 2022Recitation-Augmented Language Modelscited by 65
  19. 2023MarioGPT: Open-Ended Text2Level Generation through Large Language Modelscited by 64
  20. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 64
  21. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  22. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  23. 2024Erasing Undesirable Influence in Diffusion Modelscited by 57
  24. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 53
  25. 2024Machine Unlearning for Image-to-Image Generative Modelscited by 47
  26. 2023Bounding Training Data Reconstruction in DP-SGDcited by 46
  27. 2022Memorization in NLP Fine-tuning Methodscited by 45
  28. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 45
  29. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 43
  30. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 38
  31. 2024Data Selection via Optimal Control for Language Modelscited by 38
  32. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 37
  33. 2023Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimationcited by 36
  34. 2023User Inference Attacks on Large Language Modelscited by 36
  35. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  36. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 35
  37. 2024Large Language Models for Education: A Surveycited by 34
  38. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33

50 most cited of 220 All 220 (JSON)