mlsanityData to 7 Oct 2026

Extracting Training Data from Large Language Models

arXiv 2012.07805 ↗First posted 14 Dec 2020cs.CR cs.CL cs.LGcited by 2,144

Not on the map. Why?

Cited by 493 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.

Key facts. Cited by 2,144 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 124; 2022: 190; 2023: 354; 2024: 497; 2025: 497; 2026: 482.

2,144 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2025Gemma 3 Technical Reportcited by 1,686
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
38 more
  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  9. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  12. 2023BloombergGPT: A Large Language Model for Financecited by 923
  13. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  14. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  15. 2021Membership Inference Attacks From First Principlescited by 839
  16. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  17. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  18. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  19. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  20. 2023Extracting Training Data from Diffusion Modelscited by 723
  21. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  22. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  23. 2024A Survey on Large Language Models for Code Generationcited by 646
  24. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  25. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  26. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  27. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  28. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  29. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  30. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  31. 2021MetaICL: Learning to Learn In Contextcited by 450
  32. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  33. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  34. 2023MusicLM: Generating Music From Textcited by 442
  35. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  36. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  37. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  38. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 401

50 most cited of 2,144 All 2,144 (JSON)