mlsanityData to 7 Oct 2026

A Taxonomy for Data Contamination in Large Language Models

arXiv 2407.08716 ↗First posted 11 Jul 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Large language models pretrained on extensive web corpora demonstrate remarkable performance across a wide range of downstream tasks. However, a growing concern is data contamination, where evaluation datasets may be contained in the pretraining corpus, inflating model performance. Decontamination, the process of detecting and removing such data, is a potential solution; yet these contaminants may originate from altered versions of the test set, evading detection during decontamination. How different types of contamination impact the performance of language models on downstream tasks is not fully understood. We present a taxonomy that categorizes the various types of contamination encountered by LLMs during the pretraining phase and identify which types pose the highest risk. We analyze the impact of contamination on two key NLP tasks -- summarization and question answering -- revealing how different types of contamination influence task performance during evaluation.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

24 of 38 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  10. 2023Large Language Modelscited by 774
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  12. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
12 more
  1. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  2. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  4. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  5. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  6. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 98
  7. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  8. 2023Data Contamination Through the Lens of Timecited by 39
  9. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 30
  10. 2023Data Portraits: Recording Foundation Model Training Datacited by 29
  11. 2022Downstream Datasets Make Surprisingly Good Pretraining Corporacited by 20
  12. 2024Concerned with Data Contamination? Assessing Countermeasures in Code Language Modelcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 33
  2. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMscited by 10
  3. 2025Re-evaluating Open-ended Evaluation of Large Language Modelscited by 8
  4. 2025Policy Induction: Predicting Startup Success via Explainable Memory-Augmented In-Context Learningcited by 6
  5. 2025AdEval: Alignment-based Dynamic Evaluation to Mitigate Data Contamination in Large Language Modelscited by 1
  6. 2025LLM-AR: LLM-powered Automated Reasoning Frameworkcited by 1
  7. 2026Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implicationscited by 1
  8. 2026How Much Reasoning Do Retrieval-Augmented Models Add beyond LLMs? A Benchmarking Framework for Multi-Hop Inference over Hybrid Knowledgecited by 0
  9. 2026Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Modelscited by 0
  10. 2026Benchmark Contamination: A Taxonomy Organized by Defeated Mitigationcited by 0
  11. 2026Made in Hungary: Comments on the performance of generative language modelscited by 0