mlsanityData to 7 Oct 2026

Undesirable Memorization in Large Language Models: A Survey

arXiv 2410.02650 ↗First posted 3 Oct 2024cs.CL cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

While recent research increasingly showcases the remarkable capabilities of Large Language Models (LLMs), it is equally crucial to examine their associated risks. Among these, privacy and security vulnerabilities are particularly concerning, posing significant ethical and legal challenges. At the heart of these vulnerabilities stands memorization, which refers to a model's tendency to store and reproduce phrases from its training data. This phenomenon has been shown to be a fundamental source to various privacy and security attacks against LLMs. In this paper, we provide a taxonomy of the literature on LLM memorization, exploring it across three dimensions: granularity, retrievability, and desirability. Next, we discuss the metrics and methods used to quantify memorization, followed by an analysis of the causes and factors that contribute to memorization phenomenon. We then explore strategies that are used so far to mitigate the undesirable aspects of this phenomenon. We conclude our survey by identifying potential research topics for the near future, including methods to balance privacy and performance, and the analysis of memorization in specific LLM contexts such as conversational agents, retrieval-augmented generation, and diffusion language models. Given the rapid research pace in this field, we also maintain a dedicated repository of the references discussed in this survey which will be regularly updated to reflect the latest developments.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 117 of its 182 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

117 of 182 references.

References · 117

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
38 more
  1. 2016Deep Learning with Differential Privacycited by 2,682
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  13. 2018Hierarchical Neural Story Generationcited by 1,182
  14. 2019Machine Unlearningcited by 1,020
  15. 2019How Can We Know What Language Models Know?cited by 953
  16. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  17. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  18. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  19. 2023Extracting Training Data from Diffusion Modelscited by 723
  20. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  21. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  22. 2019Defending Against Neural Fake Newscited by 553
  23. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  24. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  25. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  26. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  27. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  28. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  29. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  30. 2022A Survey of Machine Unlearningcited by 298
  31. 2023Large Language Model Unlearningcited by 288
  32. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  33. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  34. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  35. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  36. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  37. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  38. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195

50 most cited of 117 All 117 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 11.

29 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025R.R.: Unveiling LLM Training Privacy through Recollection and Rankingcited by 10
  3. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 9
  4. 2025Mitigating Memorization in LLMs using Activation Steeringcited by 7
  5. 2025Extracting alignment data in open modelscited by 7
  6. 2025Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyondcited by 6
  7. 2025Rote Learning Considered Useful: Generalizing over Memorized Data in LLMscited by 5
  8. 2025Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implicationscited by 5
  9. 2025KnowRL: Teaching Language Models to Know What They Knowcited by 5
  10. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  11. 2026Membership Inference Attacks Against Fine-tuned Diffusion Language Modelscited by 5
  12. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
17 more
  1. 2025Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Modelscited by 3
  2. 2025Factored Agents: Decoupling In-Context Learning and Memorization for Robust Tool Usecited by 3
  3. 2025Beyond Public Access in LLM Pre-Training Datacited by 3
  4. 2025Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledgecited by 2
  5. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  6. 2025Rethinking Memorization Measures and their Implications in Large Language Modelscited by 1
  7. 2025Memorization neq Understanding: Do Large Language Models Have the Ability of Scenario Cognition?cited by 1
  8. 2025We Are All Creators: Generative AI, Collective Knowledge, and the Path Towards Human-AI Synergycited by 0
  9. 2025Comparing Reconstruction Attacks on Pretrained Versus Full Fine-tuned Large Language Model Embeddings on Homo Sapiens Splice Sites Genomic Datacited by 0
  10. 2025Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMscited by 0
  11. 2026Operationalising the Right to be Forgotten in LLMs: A Lightweight Sequential Unlearning Framework for Privacy-Aligned Deployment in Politically Sensitive Environmentscited by 0
  12. 2026Detecting Data Contamination in Large Language Modelscited by 0
  13. 2026VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generationcited by 0
  14. 2026On the Memorization Behavior of LLMs in Generative Recommendation: Observations, Implications, and Training Strategiescited by 0
  15. 2026Implicit Reasoning Steering via Concept Chainingcited by 0
  16. 2026Memorization Is Not Extraction: Tight Differential-Privacy Bounds and Audit Blind Spotscited by 0
  17. 2026Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
  2. 2026Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects
  3. 2026Unveiling Privacy Risks in Multi-modal Large Language Models: Task-specific Vulnerabilities and Mitigation Challenges
  4. 2025Analyzing Memorization in Large Language Models through the Lens of Model Attribution
  5. 2024The Mosaic Memory of Large Language Models
  6. 2024Unique Security and Privacy Threats of Large Language Models: A Comprehensive Survey
  7. 2024Understanding Privacy Risks of Embeddings Induced by Large Language Models
  8. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolution