# Data Contamination: From Memorization to Exploitation

arXiv: https://arxiv.org/abs/2203.08242 · HTML: https://mlsanity.com/paper/2203.08242/

## Key facts

- Cited by 236 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies](https://mlsanity.com/paper/2104.05694/) (2021)

## Built on by (most cited first, 30 of 236)

- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024)
- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks](https://mlsanity.com/paper/2305.10160/) (2023)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [On Memorization of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2410.23123/) (2024)
- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions](https://mlsanity.com/paper/2402.18060/) (2024)
- [Large Language Models as Analogical Reasoners](https://mlsanity.com/paper/2310.01714/) (2023)
- [Benchmarking Benchmark Leakage in Large Language Models](https://mlsanity.com/paper/2404.18824/) (2024)
- [VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use](https://mlsanity.com/paper/2308.06595/) (2023)
- [DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks](https://mlsanity.com/paper/2309.17167/) (2023)
- [InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales](https://mlsanity.com/paper/2406.13629/) (2024)
- [Task Contamination: Language Models May Not Be Few-Shot Anymore](https://mlsanity.com/paper/2312.16337/) (2023)
- [Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate](https://mlsanity.com/paper/2305.13160/) (2023)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data](https://mlsanity.com/paper/2407.14985/) (2024)
- [Eight Things to Know about Large Language Models](https://mlsanity.com/paper/2304.00612/) (2023)
- [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models](https://mlsanity.com/paper/2403.04811/) (2024)
- [MIRAGE: The Illusion of Visual Understanding](https://mlsanity.com/paper/2603.21687/) (2026)
- [HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models](https://mlsanity.com/paper/2310.16755/) (2023)

## Abstract (arXiv)

Pretrained language models are typically trained on massive web-based datasets, which are often "contaminated" with downstream test sets. It is not clear to what extent models exploit the contaminated data for downstream tasks. We present a principled method to study this question. We pretrain BERT models on joint corpora of Wikipedia and labeled downstream datasets, and fine-tune them on the relevant task. Comparing performance between samples seen and unseen during pretraining enables us to define and quantify levels of memorization and exploitation. Experiments with two models and three downstream tasks show that exploitation exists in some cases, but in others the models memorize the contaminated data, but do not exploit it. We show that these two measures are affected by different factors such as the number of duplications of the contaminated data and the model size. Our results highlight the importance of analyzing massive web-scale datasets to verify that progress in NLP is obtained by better language understanding and not better data exploitation.