mlsanityData to 14 Oct 2026

Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models

arXiv 2404.02936 ↗First posted 3 Apr 2024cs.CL cs.LGcited by 103

Not on the map. Why?

Cited by 55 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The problem of pre-training data detection for large language models (LLMs) has received growing attention due to its implications in critical issues like copyright violation and test data contamination. Despite improved performance, existing methods (including the state-of-the-art, Min-K%) are mostly developed upon simple heuristics and lack solid, reasonable foundations. In this work, we propose a novel and theoretically motivated methodology for pre-training data detection, named Min-K%++. Specifically, we present a key insight that training samples tend to be local maxima of the modeled distribution along each input dimension through maximum likelihood training, which in turn allow us to insightfully translate the problem into identification of local maxima. Then, we design our method accordingly that works under the discrete distribution modeled by LLMs, whose core idea is to determine whether the input forms a mode or has relatively high probability under the conditional categorical distribution. Empirically, the proposed method achieves new SOTA performance across multiple settings. On the WikiMIA benchmark, Min-K%++ outperforms the runner-up by 6.2% to 10.5% in detection AUROC averaged over five models. On the more challenging MIMIR benchmark, it consistently improves upon reference-free methods while performing on par with reference-based method that requires an extra reference model.

Key facts. Cited by 103 papers in the map (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

36 of 50 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  12. 2020Extracting Training Data from Large Language Modelscited by 2,189
24 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  3. 2021Membership Inference Attacks From First Principlescited by 878
  4. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  5. 2023Extracting Training Data from Diffusion Modelscited by 732
  6. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  7. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  8. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  9. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  10. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  11. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  12. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  13. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 179
  14. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 175
  15. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  16. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 147
  17. 2022The Privacy Onion Effect: Memorization is Relativecited by 136
  18. 2023Privacy Auditing with One (1) Training Runcited by 123
  19. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  20. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 119
  21. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 98
  22. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 74
  23. 2022Statistical Efficiency of Score Matching: The View from Isoperimetrycited by 64
  24. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Modelscited by 56

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 23; 2025: 53; 2026: 27.

103 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  2. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 99
  3. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  4. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 78
  5. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Modelscited by 69
  6. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 64
  7. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 63
  8. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  9. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 59
  10. 2024Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Methodcited by 55
  11. 2024Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Datacited by 50
  12. 2024Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generationcited by 44
38 more
  1. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 38
  2. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 37
  3. 2024Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignmentcited by 25
  4. 2024Context-Aware Membership Inference Attacks against Pre-trained Large Language Modelscited by 22
  5. 2024Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decodingcited by 20
  6. 2025Vid-SME: Membership Inference Attacks against Large Video Understanding Modelscited by 19
  7. 2024Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?cited by 18
  8. 2024Semantic Membership Inference Attack against Large Language Modelscited by 17
  9. 2024Beware of Calibration Data for Pruning Large Language Modelscited by 15
  10. 2025How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergencecited by 15
  11. 2025Privacy Attacks on Image AutoRegressive Modelscited by 15
  12. 2025The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contaminationcited by 15
  13. 2024Detecting Training Data of Large Language Models via Expectation Maximizationcited by 13
  14. 2025Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributionscited by 12
  15. 2025SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Designcited by 12
  16. 2025Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Papercited by 12
  17. 2025Unveiling Over-Memorization in Finetuning LLMs for Reasoning Taskscited by 11
  18. 2025Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Modelscited by 10
  19. 2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 10
  20. 2025Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Testcited by 10
  21. 2025LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Modelscited by 9
  22. 2025On The Fragility of Benchmark Contamination Detection in Reasoning Modelscited by 9
  23. 2024Fine-tuning can Help Detect Pretraining Data from Large Language Modelscited by 8
  24. 2025Synthetic Data Privacy Metricscited by 8
  25. 2025STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasingscited by 8
  26. 2025Detecting Data Contamination in LLMs via In-Context Learningcited by 8
  27. 2025Auditing Data Membership in Reinforcement Learning With Verifiable Rewardscited by 8
  28. 2025Permissioned LLMs: Enforcing Access Control in Large Language Modelscited by 7
  29. 2025Unlocking Post-hoc Dataset Inference with Synthetic Datacited by 7
  30. 2025The Surprising Effectiveness of Membership Inference with Simple N-Gram Coveragecited by 7
  31. 2026Window-based Membership Inference Attacks Against Fine-tuned Large Language Modelscited by 7
  32. 2026Is My Vision-Language Data in Your AI? Membership Inference Test (MINT) Demo 2cited by 7
  33. 2025Membership Inference Attacks on Sequence Modelscited by 6
  34. 2025Automated Detection of Pre-training Text in Black-box LLMscited by 6
  35. 2025Tag&Tab: Pretraining Data Detection in Large Language Models Using Keyword-Based Membership Inference Attackcited by 5
  36. 2026Membership Inference Attacks Against Fine-tuned Diffusion Language Modelscited by 5
  37. 2024ObfuscaTune: Obfuscated Offsite Fine-tuning and Inference of Proprietary LLMs on Private Datasetscited by 4
  38. 2025DUSK: Do Not Unlearn Shared Knowledgecited by 4

50 most cited of 103 All 103 (JSON)