mlsanityData to 14 Oct 2026

Detecting Pretraining Data from Large Language Models

arXiv 2310.16789 ↗First posted 25 Oct 2023cs.CL cs.CR cs.LGcited by 387

Not on the map. Why?

Cited by 167 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Although large language models (LLMs) are widely deployed, the data used to train them is rarely disclosed. Given the incredible scale of this data, up to trillions of tokens, it is all but certain that it includes potentially problematic text such as copyrighted materials, personally identifiable information, and test data for widely reported reference benchmarks. However, we currently have no way to know which data of these types is included or in what proportions. In this paper, we study the pretraining data detection problem: given a piece of text and black-box access to an LLM without knowing the pretraining data, can we determine if the model was trained on the provided text? To facilitate this study, we introduce a dynamic benchmark WIKIMIA that uses data created before and after model training to support gold truth detection. We also introduce a new detection method Min-K% Prob based on a simple hypothesis: an unseen example is likely to contain a few outlier words with low probabilities under the LLM, while a seen example is less likely to have words with such low probabilities. Min-K% Prob can be applied without any knowledge about the pretraining corpus or any additional training, departing from previous detection methods that require training a reference model on data that is similar to the pretraining data. Moreover, our experiments demonstrate that Min-K% Prob achieves a 7.4% improvement on WIKIMIA over these previous methods. We apply Min-K% Prob to three real-world scenarios, copyrighted book detection, contaminated downstream example detection and privacy auditing of machine unlearning, and find it a consistently effective solution.

Key facts. Cited by 387 papers in the map (data through 14 Oct 2026). Builds on 47 of its 64 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

47 of 64 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
35 more
  1. 2019Machine Unlearningcited by 1,061
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  3. 2021Membership Inference Attacks From First Principlescited by 878
  4. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  5. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  6. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  7. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  8. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  9. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  10. 2021Remember What You Want to Forget: Algorithms for Machine Unlearningcited by 371
  11. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  12. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  13. 2020Approximate Data Deletion from Machine Learning Modelscited by 255
  14. 2022Data Contamination: From Memorization to Exploitationcited by 236
  15. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 218
  16. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 199
  17. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  18. 2021Adaptive Machine Unlearningcited by 176
  19. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 175
  20. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 175
  21. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 173
  22. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  23. 2020DeltaGrad: Rapid retraining of machine learning modelscited by 149
  24. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 147
  25. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  26. 2023Privacy Auditing with One (1) Training Runcited by 123
  27. 2018Understanding Membership Inferences on Well-Generalized Learning Modelscited by 90
  28. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 90
  29. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  30. 2022Recovering Private Text in Federated Learning of Language Modelscited by 70
  31. 2021Membership Inference Attack Susceptibility of Clinical Language Modelscited by 52
  32. 2021Membership Inference on Word Embedding and Beyondcited by 35
  33. 2022Learning with Recoverable Forgettingcited by 30
  34. 2020Federated Unlearningcited by 29
  35. 2023Advancing Differential Privacy: Where We Are Now and Future Directions for Real-World Deploymentcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 117; 2025: 154; 2026: 111.

387 papers.

Sort:

By area:

  1. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  2. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  3. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
  4. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 261
  5. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  6. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  7. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  8. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  9. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  10. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
  11. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  12. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
38 more
  1. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 116
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2024LLM Dataset Inference: Did you train on my dataset?cited by 110
  4. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  5. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 106
  6. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 103
  7. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 98
  8. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 98
  9. 2024Guardrail Baselines for Unlearning in LLMscited by 96
  10. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  11. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 91
  12. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 89
  13. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 88
  14. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  15. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  16. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 83
  17. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 81
  18. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 78
  19. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  20. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Modelscited by 69
  21. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  22. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Datacited by 63
  23. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 63
  24. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 59
  25. 2024Watermarking Makes Language Models Radioactivecited by 58
  26. 2024Compression Represents Intelligence Linearlycited by 56
  27. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
  28. 2024On Large Language Model Continual Unlearningcited by 52
  29. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 51
  30. 2024Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Datacited by 50
  31. 2023Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Modelscited by 47
  32. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 47
  33. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 46
  34. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 43
  35. 2024Membership Inference Attacks Against In-Context Learningcited by 43
  36. 2024AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Textcited by 42
  37. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 42
  38. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methodscited by 40

50 most cited of 387 All 387 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023From Text to Source: Results in Detecting Large Language Model-Generated Content
  2. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods
  3. 2024Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning
  4. 2024Scalable Influence and Fact Tracing for Large Language Model Pretraining
  5. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
  6. 2025A Statistical Hypothesis Testing Framework for Data Misappropriation Detection in Large Language Models
  7. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions
  8. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls