# Undesirable Memorization in Large Language Models: A Survey

arXiv: https://arxiv.org/abs/2410.02650 · HTML: https://mlsanity.com/paper/2410.02650/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 117 of its 182 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 30 of 117)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)

## Built on by (most cited first, 29 of 29)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [R.R.: Unveiling LLM Training Privacy through Recollection and Ranking](https://mlsanity.com/paper/2502.12658/) (2025)
- [The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation](https://mlsanity.com/paper/2507.05578/) (2025)
- [Mitigating Memorization in LLMs using Activation Steering](https://mlsanity.com/paper/2503.06040/) (2025)
- [Extracting alignment data in open models](https://mlsanity.com/paper/2510.18554/) (2025)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyond](https://mlsanity.com/paper/2503.07501/) (2025)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [Rote Learning Considered Useful: Generalizing over Memorized Data in LLMs](https://mlsanity.com/paper/2507.21914/) (2025)
- [Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implications](https://mlsanity.com/paper/2509.08604/) (2025)
- [KnowRL: Teaching Language Models to Know What They Know](https://mlsanity.com/paper/2510.11407/) (2025)
- [Membership Inference Attacks Against Fine-tuned Diffusion Language Models](https://mlsanity.com/paper/2601.20125/) (2026)
- [Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models](https://mlsanity.com/paper/2503.01208/) (2025)
- [Factored Agents: Decoupling In-Context Learning and Memorization for Robust Tool Use](https://mlsanity.com/paper/2503.22931/) (2025)
- [Beyond Public Access in LLM Pre-Training Data](https://mlsanity.com/paper/2505.00020/) (2025)
- [Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledge](https://mlsanity.com/paper/2506.18998/) (2025)
- [GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning](https://mlsanity.com/paper/2603.10243/) (2026)
- [Rethinking Memorization Measures and their Implications in Large Language Models](https://mlsanity.com/paper/2507.14777/) (2025)
- [Memorization neq Understanding: Do Large Language Models Have the Ability of Scenario Cognition?](https://mlsanity.com/paper/2509.04866/) (2025)
- [We Are All Creators: Generative AI, Collective Knowledge, and the Path Towards Human-AI Synergy](https://mlsanity.com/paper/2504.07936/) (2025)
- [Comparing Reconstruction Attacks on Pretrained Versus Full Fine-tuned Large Language Model Embeddings on Homo Sapiens Splice Sites Genomic Data](https://mlsanity.com/paper/2511.07481/) (2025)
- [Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs](https://mlsanity.com/paper/2512.03310/) (2025)
- [Operationalising the Right to be Forgotten in LLMs: A Lightweight Sequential Unlearning Framework for Privacy-Aligned Deployment in Politically Sensitive Environments](https://mlsanity.com/paper/2604.12459/) (2026)
- [Detecting Data Contamination in Large Language Models](https://mlsanity.com/paper/2604.19561/) (2026)
- [VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generation](https://mlsanity.com/paper/2605.22368/) (2026)
- [On the Memorization Behavior of LLMs in Generative Recommendation: Observations, Implications, and Training Strategies](https://mlsanity.com/paper/2606.17276/) (2026)
- [Implicit Reasoning Steering via Concept Chaining](https://mlsanity.com/paper/2607.14242/) (2026)
- [Memorization Is Not Extraction: Tight Differential-Privacy Bounds and Audit Blind Spots](https://mlsanity.com/paper/2608.27782/) (2026)
- [Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes](https://mlsanity.com/paper/2610.12085/) (2026)

## Abstract (arXiv)

While recent research increasingly showcases the remarkable capabilities of Large Language Models (LLMs), it is equally crucial to examine their associated risks. Among these, privacy and security vulnerabilities are particularly concerning, posing significant ethical and legal challenges. At the heart of these vulnerabilities stands memorization, which refers to a model's tendency to store and reproduce phrases from its training data. This phenomenon has been shown to be a fundamental source to various privacy and security attacks against LLMs. In this paper, we provide a taxonomy of the literature on LLM memorization, exploring it across three dimensions: granularity, retrievability, and desirability. Next, we discuss the metrics and methods used to quantify memorization, followed by an analysis of the causes and factors that contribute to memorization phenomenon. We then explore strategies that are used so far to mitigate the undesirable aspects of this phenomenon. We conclude our survey by identifying potential research topics for the near future, including methods to balance privacy and performance, and the analysis of memorization in specific LLM contexts such as conversational agents, retrieval-augmented generation, and diffusion language models. Given the rapid research pace in this field, we also maintain a dedicated repository of the references discussed in this survey which will be regularly updated to reflect the latest developments.