# Detecting Data Contamination in LLMs via In-Context Learning

arXiv: https://arxiv.org/abs/2510.27055 · HTML: https://mlsanity.com/paper/2510.27055/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Builds on (most cited first, 30 of 36)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers](https://mlsanity.com/paper/2212.10559/) (2022)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [Certified Robustness to Adversarial Word Substitutions](https://mlsanity.com/paper/1909.00986/) (2019)
- [Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset](https://mlsanity.com/paper/2412.02595/) (2024)
- [Investigating Data Contamination in Modern Benchmarks for Large Language Models](https://mlsanity.com/paper/2311.09783/) (2023)
- [Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation](https://mlsanity.com/paper/2305.16938/) (2023)
- [LLM Dataset Inference: Did you train on my dataset?](https://mlsanity.com/paper/2406.06443/) (2024)
- [Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models](https://mlsanity.com/paper/2404.02936/) (2024)
- [Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models](https://mlsanity.com/paper/2504.03624/) (2025)
- [SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore](https://mlsanity.com/paper/2308.04430/) (2023)
- [Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?](https://mlsanity.com/paper/2402.12483/) (2024)
- [Leakage of Dataset Properties in Multi-Party Machine Learning](https://mlsanity.com/paper/2006.07267/) (2020)

## Built on by (most cited first, 8 of 8)

- [Membership Inference Attacks against Large Audio Language Models](https://mlsanity.com/paper/2603.28378/) (2026)
- [Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores](https://mlsanity.com/paper/2608.02985/) (2026)
- [HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks](https://mlsanity.com/paper/2603.12760/) (2026)
- [The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection](https://mlsanity.com/paper/2606.03305/) (2026)
- [Provable Joint Decontamination for Benchmarking Multiple Large Language Models](https://mlsanity.com/paper/2605.21543/) (2026)
- [Anchored or Drifting: What Recursive Self-Generation Reveals About Training Data](https://mlsanity.com/paper/2606.31991/) (2026)
- [PolERo: Studying Political Evasion in Romanian](https://mlsanity.com/paper/2609.02391/) (2026)
- [Structural Chain-of-Thought: Reasoning with Executable Computational Graphs](https://mlsanity.com/paper/2610.15107/) (2026)

## Abstract (arXiv)

We present Contamination Detection via Context (CoDeC), a practical and accurate method to detect and quantify training data contamination in large language models. CoDeC distinguishes between data memorized during training and data outside the training distribution by measuring how in-context learning affects model performance. We find that in-context examples typically boost confidence for unseen datasets but may reduce it when the dataset was part of training, due to disrupted memorization patterns. Experiments show that CoDeC produces interpretable contamination scores that clearly separate seen and unseen datasets, and reveals strong evidence of memorization in open-weight models with undisclosed training corpora. The method is simple, automated, and both model- and dataset-agnostic, making it easy to integrate with benchmark evaluations.