mlsanityData to 14 Oct 2026

An Open Source Data Contamination Report for Large Language Models

arXiv 2310.17589 ↗First posted 26 Oct 2023cs.CL cs.AIcited by 51

Not on the map. Why?

Abstract (arXiv)

Data contamination in model evaluation has become increasingly prevalent with the growing popularity of large language models. It allows models to "cheat" via memorisation instead of displaying true capabilities. Therefore, contamination analysis has become an crucial part of reliable model evaluation to validate results. However, existing contamination analysis is usually conducted internally by large language model developers and often lacks transparency and completeness. This paper presents an extensive data contamination report for over 15 popular large language models across six popular multiple-choice QA benchmarks. We also introduce an open-source pipeline that enables the community to perform contamination analysis on customised data and models. Our experiments reveal varying contamination levels ranging from 1% to 45% across benchmarks, with the contamination degree increasing rapidly over time. Performance analysis of large language models indicates that data contamination does not necessarily lead to increased model metrics: while significant accuracy boosts of up to 14% and 7% are observed on contaminated C-Eval and Hellaswag benchmarks, only a minimal increase is noted on contaminated MMLU. We also find larger models seem able to gain more advantages than smaller models on contaminated test sets.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 28 of its 57 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

28 of 57 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
16 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  7. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  8. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  9. 2018QuAC : Question Answering in Contextcited by 381
  10. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142
  11. 2023Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluationcited by 48
  12. 2023Data Contamination Through the Lens of Timecited by 40
  13. 2023FrameBERT: Conceptual Metaphor Detection with Frame Embedding Learningcited by 18
  14. 2023White-Box Multi-Objective Adversarial Attack on Dialogue Generationcited by 14
  15. 2023RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Modelscited by 11
  16. 2023Uncertainty-Aware Bootstrap Learning for Joint Extraction on Distantly-Supervised Datacited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 23; 2026: 17.

51 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  2. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  3. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 110
  4. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 42
  5. 2024ConStat: Performance-Based Contamination Detection in Large Language Modelscited by 33
  6. 2024Evading Data Contamination Detection for Language Models is (too) Easycited by 30
  7. 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrappingcited by 22
  8. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 21
  9. 2024Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answeringcited by 20
  10. 2025Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challengescited by 16
  11. 2024PiCO: Peer Review in LLMs based on the Consistency Optimizationcited by 14
  12. 2025VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contaminationcited by 12
38 more
  1. 2025Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributionscited by 12
  2. 2025When Do Large Language Models Exhibit Unsolicited Deception?cited by 10
  3. 2025SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questionscited by 9
  4. 2024ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Modelscited by 8
  5. 2025A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gapcited by 8
  6. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 7
  7. 2025Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluationcited by 6
  8. 2025ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoningcited by 6
  9. 2024Evaluating Large Language Models for Generalization and Robustness via Data Compressioncited by 5
  10. 2025Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluationcited by 5
  11. 2024Questionable practices in machine learningcited by 4
  12. 2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learningcited by 3
  13. 2025Controlling the Risk of Corrupted Contexts for Language Models via Early-Exitingcited by 3
  14. 2025Risk Management for Mitigating Benchmark Failure Modes: BenchRiskcited by 3
  15. 2025WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorizationcited by 2
  16. 2025A Benchmark for End-to-End Zero-Shot Biomedical Relation Extraction with LLMs: Experiments with OpenAI Modelscited by 2
  17. 2025MACEval: A Multi-Agent Continual Evaluation Network for Large Modelscited by 2
  18. 2026Quantifying the Effect of Test Set Contamination on Generative Evaluationscited by 2
  19. 2026CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization Vcited by 2
  20. 2026LLM Benchmark Datasets Should Be Contamination-Resistantcited by 2
  21. 2025AdEval: Alignment-based Dynamic Evaluation to Mitigate Data Contamination in Large Language Modelscited by 1
  22. 2025DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometrycited by 1
  23. 2026Is It Novel and Why? Fine-Grained Patent Novelty Prediction Based on Passage Retrievalcited by 1
  24. 2026Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluationcited by 1
  25. 2025Chat-of-Thought: Collaborative Multi-Agent System for Generating Domain Specific Informationcited by 0
  26. 2025Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMscited by 0
  27. 2025Citation Failure: Definition, Analysis and Efficient Mitigationcited by 0
  28. 2026Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMscited by 0
  29. 2026When AI Navigates the Fog of Warcited by 0
  30. 2026SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarkscited by 0
  31. 2026Can Large Language Models Reliably Correct Errors in Low-Resource ASR? A Contamination-Aware Case Study on West Frisiancited by 0
  32. 2026Provable Joint Decontamination for Benchmarking Multiple Large Language Modelscited by 0
  33. 2026Correcting test set contamination by spiking the training datacited by 0
  34. 2026Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Modelscited by 0
  35. 2026LeakScale: Estimating the Causal Effect of Benchmark Exposurecited by 0
  36. 2026Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Modelscited by 0
  37. 2026Metric-Construction Coupling Inflates Measured Synthetic Dialect Recoverycited by 0
  38. 2026Video-Arena: A Scalable Agentic Video Benchmark for Spatiotemporal Capabilitycited by 0

50 most cited of 51 All 51 (JSON)