mlsanityData to 7 Oct 2026

Scalable Extraction of Training Data from (Production) Language Models

arXiv 2311.17035 ↗First posted 28 Nov 2023cs.LG cs.CL cs.CRcited by 415

Not on the map. Why?

Cited by 120 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open models like LLaMA or Falcon, and closed models like ChatGPT. Existing techniques from the literature suffice to attack unaligned models; in order to attack the aligned ChatGPT, we develop a new divergence attack that causes the model to diverge from its chatbot-style generations and emit training data at a rate 150x higher than when behaving properly. Our methods show practical attacks can recover far more data than previously thought, and reveal that current alignment techniques do not eliminate memorization.

Key facts. Cited by 415 papers in the map (data through 7 Oct 2026). Builds on 8 of its 56 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

8 of 56 references.

References · 8

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2023Scaling Data-Constrained Language Modelscited by 391
  7. 2023Are aligned neural networks adversarially aligned?cited by 305
  8. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 206; 2025: 125; 2026: 81.

415 papers.

Sort:

By area:

  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  2. 2025Gemma 3 Technical Reportcited by 1,686
  3. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  4. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  5. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  6. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  7. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  8. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  9. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  10. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  11. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  12. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
38 more
  1. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  2. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  3. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  4. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  5. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  6. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  8. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  9. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  10. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 95
  11. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  12. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  13. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  14. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  15. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 90
  16. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  17. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  18. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  19. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  20. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 85
  21. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  22. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  23. 2024Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacycited by 78
  24. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  25. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 72
  26. 2024Large language models surpass human experts in predicting neuroscience resultscited by 69
  27. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  28. 2024Instructional Fingerprinting of Large Language Modelscited by 64
  29. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  30. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  31. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 61
  32. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Benchcited by 61
  33. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  34. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  35. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  36. 2024LLMmap: Fingerprinting For Large Language Modelscited by 56
  37. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  38. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54

50 most cited of 415 All 415 (JSON)