# Evading Data Contamination Detection for Language Models is (too) Easy

arXiv: https://arxiv.org/abs/2402.02823 · HTML: https://mlsanity.com/paper/2402.02823/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 52 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Builds on (most cited first, 30 of 37)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks](https://mlsanity.com/paper/2203.03929/) (2022)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [On the Importance of Difficulty Calibration in Membership Inference Attacks](https://mlsanity.com/paper/2111.08440/) (2021)
- [Investigating Data Contamination in Modern Benchmarks for Large Language Models](https://mlsanity.com/paper/2311.09783/) (2023)
- [Task Contamination: Language Models May Not Be Few-Shot Anymore](https://mlsanity.com/paper/2312.16337/) (2023)
- [Paraphrase Generation with Deep Reinforcement Learning](https://mlsanity.com/paper/1711.00279/) (2017)
- [LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction](https://mlsanity.com/paper/2312.12343/) (2023)

## Built on by (most cited first, 30 of 30)

- [A Survey on Data Contamination for Large Language Models](https://mlsanity.com/paper/2502.14425/) (2025)
- [ConStat: Performance-Based Contamination Detection in Large Language Models](https://mlsanity.com/paper/2405.16281/) (2024)
- [Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation](https://mlsanity.com/paper/2406.14644/) (2024)
- [VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbation](https://mlsanity.com/paper/2406.17681/) (2024)
- [Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions](https://mlsanity.com/paper/2410.18966/) (2024)
- [Language Models May Verbatim Complete Text They Were Not Explicitly Trained On](https://mlsanity.com/paper/2503.17514/) (2025)
- [TabEBM: A Tabular Data Augmentation Method with Distinct Class-Specific Energy-Based Models](https://mlsanity.com/paper/2409.16118/) (2024)
- [Ward: Provable RAG Dataset Inference via LLM Watermarks](https://mlsanity.com/paper/2410.03537/) (2024)
- [The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination](https://mlsanity.com/paper/2503.16402/) (2025)
- [LastingBench: Defend Benchmarks Against Knowledge Leakage](https://mlsanity.com/paper/2506.21614/) (2025)
- [LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation](https://mlsanity.com/paper/2412.10424/) (2024)
- [A Comprehensive Survey of Contamination Detection Methods in Large Language Models](https://mlsanity.com/paper/2404.00699/) (2024)
- [DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoning](https://mlsanity.com/paper/2406.04197/) (2024)
- [On The Fragility of Benchmark Contamination Detection in Reasoning Models](https://mlsanity.com/paper/2510.02386/) (2025)
- [Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation](https://mlsanity.com/paper/2512.20908/) (2025)
- [LSHBloom: Memory-efficient, Extreme-scale Document Deduplication](https://mlsanity.com/paper/2411.04257/) (2024)
- [Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation](https://mlsanity.com/paper/2409.00696/) (2024)
- [Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis](https://mlsanity.com/paper/2506.04142/) (2025)
- [CoreEval: Automatically Building Contamination-Resilient Datasets with Real-World Knowledge toward Reliable LLM Evaluation](https://mlsanity.com/paper/2511.18889/) (2025)
- [The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation](https://mlsanity.com/paper/2605.21856/) (2026)
- [LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models](https://mlsanity.com/paper/2508.05452/) (2025)
- [ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluation](https://mlsanity.com/paper/2510.08569/) (2025)
- [CAP: Data Contamination Detection via Consistency Amplification](https://mlsanity.com/paper/2410.15005/) (2024)
- [AdEval: Alignment-based Dynamic Evaluation to Mitigate Data Contamination in Large Language Models](https://mlsanity.com/paper/2501.13983/) (2025)
- [When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors](https://mlsanity.com/paper/2604.21255/) (2026)
- [TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs](https://mlsanity.com/paper/2605.24079/) (2026)
- [Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implications](https://mlsanity.com/paper/2605.26133/) (2026)
- [Learning to Detect Language Model Training Data via Active Reconstruction](https://mlsanity.com/paper/2602.19020/) (2026)
- [When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits](https://mlsanity.com/paper/2608.07914/) (2026)
- [CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds](https://mlsanity.com/paper/2609.22183/) (2026)

## Abstract (arXiv)

Large language models are widespread, with their performance on benchmarks frequently guiding user preferences for one model over another. However, the vast amount of data these models are trained on can inadvertently lead to contamination with public benchmarks, thus compromising performance measurements. While recently developed contamination detection methods try to address this issue, they overlook the possibility of deliberate contamination by malicious model providers aiming to evade detection. We argue that this setting is of crucial importance as it casts doubt on the reliability of public benchmarks. To more rigorously study this issue, we propose a categorization of both model providers and contamination detection methods. This reveals vulnerabilities in existing methods that we exploit with EAL, a simple yet effective contamination technique that significantly inflates benchmark performance while completely evading current detection methods.