mlsanityData to 7 Oct 2026

Bag of Tricks for Training Data Extraction from Language Models

arXiv 2302.04460 ↗First posted 9 Feb 2023cs.CL cs.AI cs.CRcited by 67

Not on the map. Why?

Cited by 23 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the advance of language models, privacy protection is receiving more attention. Training data extraction is therefore of great importance, as it can serve as a potential tool to assess privacy leakage. However, due to the difficulty of this task, most of the existing methods are proof-of-concept and still not effective enough. In this paper, we investigate and benchmark tricks for improving training data extraction using a publicly available dataset. Because most existing extraction methods use a pipeline of generating-then-ranking, i.e., generating text candidates as potential training data and then ranking them based on specific criteria, our research focuses on the tricks for both text generation (e.g., sampling strategy) and text ranking (e.g., token-level criteria). The experimental results show that several previously overlooked tricks can be crucial to the success of training data extraction. Based on the GPT-Neo 1.3B evaluation results, our proposed tricks outperform the baseline by a large margin in most cases, providing a much stronger baseline for future research. The code is available at https://github.com/weichen-yu/LM-Extraction.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

37 of 62 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2017A Closer Look at Memorization in Deep Networkscited by 936
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
25 more
  1. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  2. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  4. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
  5. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  6. 2020Label-Only Membership Inference Attackscited by 371
  7. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  8. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  9. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  10. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  11. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  12. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 168
  13. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 159
  14. 2019Reconstruction and Membership Inference Attacks against Generative Modelscited by 122
  15. 2018Learning to Write with Cooperative Discriminatorscited by 119
  16. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  17. 2022Locally Typical Samplingcited by 108
  18. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  19. 2020Model Extraction Attacks on Graph Neural Networks: Taxonomy and Realizationcited by 39
  20. 2016You are Who You Know and How You Behave: Attribute Inference Attacks via Users' Social Friends and Behaviorscited by 34
  21. 2020Against Membership Inference Attack: Pruning is All You Needcited by 32
  22. 2019Segmentations-Leak: Membership Inference Attacks and Defenses in Semantic Image Segmentationcited by 29
  23. 2018Autotune: A Derivative-free Optimization Framework for Hyperparameter Tuningcited by 20
  24. 2019Alleviating Privacy Attacks via Causal Learningcited by 19
  25. 2021Property Inference Attacks on Convolutional Neural Networks: Influence and Implications of Target Model's Complexitycited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 24; 2025: 21; 2026: 13.

67 papers.

Sort:

By area:

  1. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  2. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  3. 2023DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Textcited by 127
  4. 2023Foundation Models and Fair Usecited by 112
  5. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  6. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  7. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  8. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  9. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  10. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  11. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
  12. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
38 more
  1. 2023Towards Machine Unlearning Benchmarks: Forgetting the Personal Identities in Facial Recognition Systemscited by 27
  2. 2023MoPe: Model Perturbation-based Privacy Attacks on Language Modelscited by 25
  3. 2024Purifying Large Language Models by Ensembling a Small Language Modelcited by 25
  4. 2025RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patternscited by 24
  5. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  6. 2024Avoiding Copyright Infringement via Large Language Model Unlearningcited by 22
  7. 2024PII-Compass: Guiding LLM training data extraction prompts towards the target PII via groundingcited by 22
  8. 2024Sampling-based Pseudo-Likelihood for Membership Inference Attackscited by 18
  9. 2024Poison-splat: Computation Cost Attack on 3D Gaussian Splattingcited by 17
  10. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  11. 2023PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learnerscited by 13
  12. 2023Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Modelscited by 12
  13. 2024Privacy-Aware Visual Language Modelscited by 12
  14. 2025Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1Mcited by 12
  15. 2024SoK: Reducing the Vulnerability of Fine-tuned Language Models to Membership Inference Attackscited by 10
  16. 2025R.R.: Unveiling LLM Training Privacy through Recollection and Rankingcited by 10
  17. 2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 9
  18. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 9
  19. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  20. 2024Distribution-Level Feature Distancing for Machine Unlearning: Towards a Better Trade-off Between Model Utility and Forgettingcited by 7
  21. 2025Mitigating Memorization in LLMs using Activation Steeringcited by 7
  22. 2025Be the Change You Want to See: Revisiting Remote Sensing Change Detection Practicescited by 6
  23. 2025Automated Detection of Pre-training Text in Black-box LLMscited by 5
  24. 2024Promise and Peril of Collaborative Code Generation Models: Balancing Effectiveness and Memorizationcited by 4
  25. 2026AttenMIA: LLM Membership Inference Attack through Attention Signalscited by 4
  26. 2024Multi-PA: A Multi-perspective Benchmark on Privacy Assessment for Large Vision-Language Modelscited by 3
  27. 2024Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspapercited by 2
  28. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  29. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  30. 2024On Active Privacy Auditing in Supervised Fine-tuning for White-Box Language Modelscited by 1
  31. 2025AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMscited by 1
  32. 2025Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Modelscited by 1
  33. 2025Efficient Unlearning with Privacy Guaranteescited by 1
  34. 2025Guess or Recall? Training CNNs to Classify and Localize Memorization in LLMscited by 1
  35. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  36. 2025ToFU: Transforming How Federated Learning Systems Forget User Datacited by 1
  37. 2025Black Box Absorption: LLMs Undermining Innovative Ideascited by 1
  38. 2026A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Modelscited by 1

50 most cited of 67 All 67 (JSON)