mlsanityData to 14 Oct 2026

Training Data Extraction From Pre-trained Language Models: A Survey

arXiv 2305.16157 ↗First posted 25 May 2023cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

As the deployment of pre-trained language models (PLMs) expands, pressing security concerns have arisen regarding the potential for malicious extraction of training data, posing a threat to data privacy. This study is the first to provide a comprehensive survey of training data extraction from PLMs. Our review covers more than 100 key papers in fields such as natural language processing and security. First, preliminary knowledge is recapped and a taxonomy of various definitions of memorization is presented. The approaches for attack and defense are then systemized. Furthermore, the empirical findings of several quantitative studies are highlighted. Finally, future research directions based on this review are suggested.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 99 of its 146 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

99 of 146 references.

References · 99

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2019Advances and Open Problems in Federated Learningcited by 3,046
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
38 more
  1. 2016Deep Learning with Differential Privacycited by 2,728
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2018Hierarchical Neural Story Generationcited by 1,193
  9. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  10. 2022Red Teaming Language Models with Language Modelscited by 1,026
  11. 2023BloombergGPT: A Large Language Model for Financecited by 939
  12. 2016Deep Variational Information Bottleneckcited by 930
  13. 2021Membership Inference Attacks From First Principlescited by 878
  14. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  15. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  16. 2022Galactica: A Large Language Model for Sciencecited by 744
  17. 2023Extracting Training Data from Diffusion Modelscited by 732
  18. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  19. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  20. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  21. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  22. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  23. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 520
  24. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  25. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  26. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  27. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  28. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  29. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  30. 2020Label-Only Membership Inference Attackscited by 388
  31. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
  32. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 321
  33. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  34. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 292
  35. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  36. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  37. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 246
  38. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 20; 2025: 11; 2026: 7.

45 papers.

Sort:

By area:

  1. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158
  2. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  3. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 82
  4. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 71
  5. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  6. 2023Reducing Privacy Risks in Online Self-Disclosures with Language Modelscited by 47
  7. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 37
  8. 2025PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliancecited by 37
  9. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 37
  10. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  11. 2023Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Datacited by 33
  12. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
33 more
  1. 2023Traces of Memorisation in Large Language Models for Codecited by 29
  2. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  3. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  4. 2024Privacy Preserving Prompt Engineering: A Surveycited by 25
  5. 2024Data Contamination Calibration for Black-box LLMscited by 24
  6. 2024PII-Compass: Guiding LLM training data extraction prompts towards the target PII via groundingcited by 24
  7. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  8. 2024Sampling-based Pseudo-Likelihood for Membership Inference Attackscited by 20
  9. 2024Causal Estimation of Memorisation Profilescited by 20
  10. 2024Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?cited by 13
  11. 2024Do LLMs Know to Respect Copyright Notice?cited by 13
  12. 2024LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Surveycited by 12
  13. 2025VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contaminationcited by 12
  14. 2024Pandora's White-Box: Precise Training Data Detection and Extraction in Large Language Modelscited by 11
  15. 2024Replication in Visual Diffusion Models: A Survey and Outlookcited by 11
  16. 2025Automated Privacy Information Annotation in Large Language Model Interactionscited by 6
  17. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  18. 2026AttenMIA: LLM Membership Inference Attack through Attention Signalscited by 4
  19. 2024Do LLMs Dream of Ontologies?cited by 3
  20. 2024Protecting Copyrighted Material with Unique Identifiers in Large Language Model Trainingcited by 3
  21. 2025How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuningcited by 3
  22. 2024Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspapercited by 2
  23. 2024A Theoretical Survey on Foundation Modelscited by 1
  24. 2026Orchestration-Free Customer Service Automation: A Privacy-Preserving and Flowchart-Guided Frameworkcited by 1
  25. 2026Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implicationscited by 1
  26. 2025Evidencing Unauthorized Training Data from AI Generated Content using Information Isotopescited by 0
  27. 2025Current State in Privacy-Preserving Text Preprocessing for Domain-Agnostic NLPcited by 0
  28. 2025Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniquenesscited by 0
  29. 2025Fast-MIA: Efficient and Scalable Membership Inference for LLMscited by 0
  30. 2026Channel-Level Semantic Perturbations: Unlearnable Examples for Diverse Training Paradigmscited by 0
  31. 2026Selective Token-Level Cryptographic Redaction for Privacy-Preserving Clinical Deployment of Large Language Modelscited by 0
  32. 2026The Privacy-Hallucination Tradeoff in Differentially Private Language Modelscited by 0
  33. 2026Tokenized Key-Gated Adapter Routing: A Secure Access Control Mechanism Against Private Data Leakage in LLMscited by 0