mlsanityData to 7 Oct 2026

Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation

arXiv 2406.14644 ↗First posted 20 Jun 2024cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Data contamination has garnered increased attention in the era of large language models (LLMs) due to the reliance on extensive internet-derived training corpora. The issue of training corpus overlap with evaluation benchmarks--referred to as contamination--has been the focus of significant recent research. This body of work aims to identify contamination, understand its impacts, and explore mitigation strategies from diverse perspectives. However, comprehensive studies that provide a clear pathway from foundational concepts to advanced insights are lacking in this nascent field. Therefore, we present a comprehensive survey in the field of data contamination, laying out the key issues, methodologies, and findings to date, and highlighting areas in need of further research and development. In particular, we begin by examining the effects of data contamination across various stages and forms. We then provide a detailed analysis of current contamination detection methods, categorizing them to highlight their focus, assumptions, strengths, and limitations. We also discuss mitigation strategies, offering a clear guide for future research. This survey serves as a succinct overview of the most recent advancements in data contamination research, providing a straightforward guide for the benefit of future research endeavors.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 51 of its 66 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

51 of 66 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
38 more
  1. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  2. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2021Membership Inference Attacks From First Principlescited by 839
  6. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  7. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  8. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  9. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  10. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  11. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  12. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  13. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  14. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  15. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  16. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  17. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  18. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  19. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 98
  20. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  21. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  22. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 72
  23. 2023Can we trust the evaluation on ChatGPT?cited by 63
  24. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  25. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 62
  26. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  27. 2021Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representationscited by 60
  28. 2023LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Constructioncited by 57
  29. 2023Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluationcited by 47
  30. 2023Data Contamination Through the Lens of Timecited by 39
  31. 2023Pretraining on the Test Set Is All You Needcited by 33
  32. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 30
  33. 2024Evading Data Contamination Detection for Language Models is (too) Easycited by 30
  34. 2024Data Contamination Calibration for Black-box LLMscited by 23
  35. 2024Critical Data Size of Language Models from a Grokking Perspectivecited by 22
  36. 2024Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translationcited by 21
  37. 2022FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relationscited by 20
  38. 2024Elephants Never Forget: Testing Language Models for Memorization of Tabular Datacited by 19

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 13.

30 papers.

Sort:

By area:

  1. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  2. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  3. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  4. 2025A Survey on Data Contamination for Large Language Modelscited by 33
  5. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  6. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  7. 2024FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documentscited by 19
  8. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problemscited by 17
  9. 2025Search-Time Data Contaminationcited by 16
  10. 2025DCR: Quantifying Data Contamination in LLMs Evaluationcited by 10
  11. 2025Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Paperscited by 9
  12. 2026From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Levelcited by 6
18 more
  1. 2026When Benchmarks Leak: Inference-Time Decontamination for LLMscited by 5
  2. 2026Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wildcited by 5
  3. 2026Internal Data Repetition Destroys Language Modelscited by 3
  4. 2025Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Modelscited by 2
  5. 2026AI chatbots versus human healthcare professionals: a systematic review and meta-analysis of empathy in patient carecited by 2
  6. 2026LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detectioncited by 2
  7. 2025AdEval: Alignment-based Dynamic Evaluation to Mitigate Data Contamination in Large Language Modelscited by 1
  8. 2025Multilingual Definition Modelingcited by 1
  9. 2025Text-to-SQL Task-oriented Dialogue Ontology Constructioncited by 1
  10. 2025Graph-Based Alternatives to LLMs for Human Simulationcited by 1
  11. 2025Seeing Straight: Document Orientation Detection for Efficient OCRcited by 1
  12. 2026Quantifying the Effect of Test Set Contamination on Generative Evaluationscited by 1
  13. 2026Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implicationscited by 1
  14. 2026Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneitycited by 0
  15. 2026OpenAnt: LLM-Powered Vulnerability Discovery Through Code Decomposition, Adversarial Verification, and Dynamic Testingcited by 0
  16. 2026Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contaminationcited by 0
  17. 2026LeakScale: Estimating the Causal Effect of Benchmark Exposurecited by 0
  18. 2026MAADBench: The Refreshable Paradigm for Anomaly Detection in Multi-Agent Systemscited by 0