mlsanityData to 13 Oct 2026

Proving Test Set Contamination in Black Box Language Models

arXiv 2310.17623 ↗First posted 26 Oct 2023cs.CL cs.LGcited by 228

Not on the map. Why?

Cited by 24 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models are trained on vast amounts of internet data, prompting concerns and speculation that they have memorized public benchmarks. Going from speculation to proof of contamination is challenging, as the pretraining data used by proprietary models are often not publicly accessible. We show that it is possible to provide provable guarantees of test set contamination in language models without access to pretraining data or model weights. Our approach leverages the fact that when there is no data contamination, all orderings of an exchangeable benchmark should be equally likely. In contrast, the tendency for language models to memorize example order means that a contaminated language model will find certain canonical orderings to be much more likely than others. Our test flags potential contamination whenever the likelihood of a canonically ordered benchmark dataset is significantly higher than the likelihood after shuffling the examples. We demonstrate that our procedure is sensitive enough to reliably prove test set contamination in challenging situations, including models as small as 1.4 billion parameters, on small test sets of only 1000 examples, and datasets that appear only a few times in the pretraining corpus. Using our test, we audit five popular publicly accessible language models for test set contamination and find little evidence for pervasive contamination.

Key facts. Cited by 228 papers in the map (data through 13 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

27 of 38 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,676
15 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,273
  3. 2020Extracting Training Data from Large Language Modelscited by 2,181
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,163
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,133
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 973
  7. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  8. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  9. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 564
  10. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 485
  11. 2023Detecting Pretraining Data from Large Language Modelscited by 383
  12. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  13. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 255
  14. 2022Data Contamination: From Memorization to Exploitationcited by 236
  15. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 176

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 7; 2024: 86; 2025: 71; 2026: 64.

228 papers.

Sort:

By area:

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,163
  3. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 250
  4. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 201
  5. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  6. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 194
  7. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 137
  8. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  9. 2024On Memorization of Large Language Models in Logical Reasoningcited by 115
  10. 2024LLM Dataset Inference: Did you train on my dataset?cited by 110
  11. 2024Best Practices and Lessons Learned on Synthetic Datacited by 105
  12. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 105
38 more
  1. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  2. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 99
  3. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  4. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  5. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 73
  6. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 70
  7. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 69
  8. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Modelscited by 67
  9. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 65
  10. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 63
  11. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  12. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 57
  13. 2024Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Methodcited by 55
  14. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  15. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  16. 2023Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Modelscited by 47
  17. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 47
  18. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 43
  19. 2024Proving membership in LLM pretraining data via data watermarkscited by 40
  20. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  21. 2024OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scalecited by 37
  22. 2024KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Modelscited by 36
  23. 2024ConStat: Performance-Based Contamination Detection in Large Language Modelscited by 33
  24. 2024Membership Inference Attacks against Large Vision-Language Modelscited by 32
  25. 2024Examining the robustness of LLM evaluation to the distributional assumptions of benchmarkscited by 31
  26. 2024Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?cited by 31
  27. 2024Evading Data Contamination Detection for Language Models is (too) Easycited by 30
  28. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  29. 2024RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Contentcited by 26
  30. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  31. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  32. 2025Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generationcited by 26
  33. 2024Probing Language Models for Pre-training Data Detectioncited by 24
  34. 2024Training on the Benchmark Is Not All You Needcited by 24
  35. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  36. 2024Generative Monoculture in Large Language Modelscited by 22
  37. 2024The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?cited by 22
  38. 2025Language Models May Verbatim Complete Text They Were Not Explicitly Trained Oncited by 21

50 most cited of 228 All 228 (JSON)