# Quantifying and Analyzing Entity-level Memorization in Large Language Models

arXiv: https://arxiv.org/abs/2308.15727 · HTML: https://mlsanity.com/paper/2308.15727/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 30 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [Counterfactual Memorization in Neural Language Models](https://mlsanity.com/paper/2112.12938/) (2021)
- [Analyzing Information Leakage of Updates to Natural Language Models](https://mlsanity.com/paper/1912.07942/) (2019)
- [Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning](https://mlsanity.com/paper/2305.11759/) (2023)
- [Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakage](https://mlsanity.com/paper/2305.12707/) (2023)

## Built on by (most cited first, 18 of 18)

- [Undesirable Memorization in Large Language Models: A Survey](https://mlsanity.com/paper/2410.02650/) (2024)
- [The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggers](https://mlsanity.com/paper/2602.03085/) (2026)
- [Pandora's White-Box: Precise Training Data Detection and Extraction in Large Language Models](https://mlsanity.com/paper/2402.17012/) (2024)
- [The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation](https://mlsanity.com/paper/2507.05578/) (2025)
- [The Fault in our Stars: Quality Assessment of Code Generation Benchmarks](https://mlsanity.com/paper/2404.10155/) (2024)
- [Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2408.04643/) (2024)
- [Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis](https://mlsanity.com/paper/2505.03019/) (2025)
- [What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests](https://mlsanity.com/paper/2507.11128/) (2025)
- [Do LLMs Dream of Ontologies?](https://mlsanity.com/paper/2401.14931/) (2024)
- [A Lightweight Method to Disrupt Memorized Sequences in LLM](https://mlsanity.com/paper/2502.05159/) (2025)
- [Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know?](https://mlsanity.com/paper/2506.02058/) (2025)
- [Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2506.06060/) (2025)
- [Data Compressibility Quantifies LLM Memorization](https://mlsanity.com/paper/2507.06056/) (2025)
- [Rethinking Memorization Measures and their Implications in Large Language Models](https://mlsanity.com/paper/2507.14777/) (2025)
- [The End Of Universal Lifelong Identifiers: Identity Systems For The AI Era](https://mlsanity.com/paper/2506.02027/) (2025)
- [Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniqueness](https://mlsanity.com/paper/2510.10902/) (2025)
- [EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMs](https://mlsanity.com/paper/2511.00192/) (2025)
- [A Privacy by Design Framework for Large Language Model-Based Applications for Children](https://mlsanity.com/paper/2602.17418/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been proven capable of memorizing their training data, which can be extracted through specifically designed prompts. As the scale of datasets continues to grow, privacy risks arising from memorization have attracted increasing attention. Quantifying language model memorization helps evaluate potential privacy risks. However, prior works on quantifying memorization require access to the precise original data or incur substantial computational overhead, making it difficult for applications in real-world language models. To this end, we propose a fine-grained, entity-level definition to quantify memorization with conditions and metrics closer to real-world scenarios. In addition, we also present an approach for efficiently extracting sensitive entities from autoregressive language models. We conduct extensive experiments based on the proposed, probing language models' ability to reconstruct sensitive entities under different settings. We find that language models have strong memorization at the entity level and are able to reproduce the training data even with partial leakages. The results demonstrate that LLMs not only memorize their training data but also understand associations between entities. These findings necessitate that trainers of LLMs exercise greater prudence regarding model memorization, adopting memorization mitigation techniques to preclude privacy violations.