mlsanityData to 7 Oct 2026

Measuring memorization in language models via probabilistic extraction

arXiv 2410.19482 ↗First posted 25 Oct 2024cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are susceptible to memorizing training data, raising concerns about the potential extraction of sensitive information at generation time. Discoverable extraction is the most common method for measuring this issue: split a training example into a prefix and suffix, then prompt the LLM with the prefix, and deem the example extractable if the LLM generates the matching suffix using greedy sampling. This definition yields a yes-or-no determination of whether extraction was successful with respect to a single query. Though efficient to compute, we show that this definition is unreliable because it does not account for non-determinism present in more realistic (non-greedy) sampling schemes, for which LLMs produce a range of outputs for the same prompt. We introduce probabilistic discoverable extraction, which, without additional cost, relaxes discoverable extraction by considering multiple queries to quantify the probability of extracting a target sequence. We evaluate our probabilistic measure across different models, sampling schemes, and training-data repetitions, and find that this measure provides more nuanced information about extraction risk compared to traditional discoverable extraction.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 24 of its 40 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

24 of 40 references.

References · 24

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  12. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
12 more
  1. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  2. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  3. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  4. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  5. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  6. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  7. 2022Defending against Reconstruction Attacks with R'enyi Differential Privacycited by 30
  8. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  9. 2024PII-Compass: Guiding LLM training data extraction prompts towards the target PII via groundingcited by 22
  10. 2024Unlocking Memorization in Large Language Models with Dynamic Soft Promptingcited by 21
  11. 2023Report of the 1st Workshop on Generative AI and Lawcited by 12
  12. 2024Extracting Memorized Training Data via Decompositioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 22.

40 papers.

Sort:

By area:

  1. 2026Extracting books from production language modelscited by 16
  2. 2026Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teamingcited by 15
  3. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 9
  4. 2025Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generationcited by 8
  5. 2025Extracting alignment data in open modelscited by 7
  6. 2025Mitigating Unintended Memorization with LoRA in Federated Learning for LLMscited by 6
  7. 2025Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Modelscited by 6
  8. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  9. 2025What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requestscited by 4
  10. 2026Reliable Graph-RAG for Codebases: AST-Derived Graphs vs LLM-Extracted Knowledge Graphscited by 4
  11. 2025Retracing the Past: LLMs Emit Training Data When They Get Lostcited by 3
  12. 2026Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effectscited by 3
28 more
  1. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  2. 2026Memorization Dynamics in Knowledge Distillation for Language Modelscited by 2
  3. 2026Caliper: Probing Lexical Anchors versus Causal Structure in LLMscited by 2
  4. 2025(Token-Level) InfoRMIA: Stronger Membership Inference and Memorization Assessment for LLMscited by 1
  5. 2026Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMscited by 1
  6. 2026Beyond Indistinguishability: Measuring Extraction Risk in LLM APIscited by 1
  7. 2026Decomposing Memorization Reduction in Privacy-Preserving Fine-Tuning of SLMs for CSIRTscited by 1
  8. 2026Can LLMs Discover Scientific Laws in Real and Parallel Worlds?cited by 1
  9. 2025Counterfactual Influence as a Distributional Quantitycited by 0
  10. 2025Are Humans as Brittle as Large Language Models?cited by 0
  11. 2025Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniquenesscited by 0
  12. 2025Memory-based Language Models: An Efficient, Explainable, and Eco-friendly Approach to Large Language Modelingcited by 0
  13. 2025PrivacyGuard: A Modular Framework for Privacy Auditing in Machine Learningcited by 0
  14. 2025EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMscited by 0
  15. 2025Remembering Unequally: Global and Disciplinary Bias in LLM Reconstruction of Scholarly Coauthor Listscited by 0
  16. 2025Towards Privacy-Preserving Code Generation: Differentially Private Code Language Modelscited by 0
  17. 2026Learning to Detect Language Model Training Data via Active Reconstructioncited by 0
  18. 2026A Comparative Analysis of LLM Memorization at Statistical and Internal Levels: Cross-Model Commonalities and Model-Specific Signaturescited by 0
  19. 2026Memorization Dynamics of Fill-in-the-Middle Pretrainingcited by 0
  20. 2026Extracting Training Data from Diffusion Language Models via Infillingcited by 0
  21. 2026POLARIS: Guiding Small Models to Write Long Storiescited by 0
  22. 2026Extractable Memorization From First Principlescited by 0
  23. 2026Leak It: Per-Document Extraction Beyond Aggregate Membership Inferencecited by 0
  24. 2026Private Direct Preference Optimization for LLM Alignmentcited by 0
  25. 2026What to Forget in Unlearning? Forget Set Curation for Language Modelscited by 0
  26. 2026Memorization Is Not Extraction: Tight Differential-Privacy Bounds and Audit Blind Spotscited by 0
  27. 2026Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetrycited by 0
  28. 2026Not All 4-bit Quantizers Are Equal: Deployment-Time Mitigation of PII Leakage in Fine-Tuned Small Language Modelscited by 0