mlsanityData to 14 Oct 2026

Detecting Data Contamination in LLMs via In-Context Learning

arXiv 2510.27055 ↗First posted 30 Oct 2025cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

We present Contamination Detection via Context (CoDeC), a practical and accurate method to detect and quantify training data contamination in large language models. CoDeC distinguishes between data memorized during training and data outside the training distribution by measuring how in-context learning affects model performance. We find that in-context examples typically boost confidence for unseen datasets but may reduce it when the dataset was part of training, due to disrupted memorization patterns. Experiments show that CoDeC produces interpretable contamination scores that clearly separate seen and unseen datasets, and reveals strong evidence of memorization in open-weight models with undisclosed training corpora. The method is simple, automated, and both model- and dataset-agnostic, making it easy to integrate with benchmark evaluations.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

36 of 45 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  6. 2020Extracting Training Data from Large Language Modelscited by 2,189
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  9. 2022Transformers learn in-context by gradient descentcited by 732
  10. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  11. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  12. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
24 more
  1. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  2. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  3. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  4. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  5. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  6. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
  7. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  8. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  9. 2019Certified Robustness to Adversarial Word Substitutionscited by 168
  10. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  11. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  12. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  13. 2024LLM Dataset Inference: Did you train on my dataset?cited by 110
  14. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 103
  15. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 100
  16. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  17. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 71
  18. 2020Leakage of Dataset Properties in Multi-Party Machine Learningcited by 47
  19. 2024Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?cited by 32
  20. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 30
  21. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  22. 2023Interaction-level Membership Inference Attack Against Federated Recommender Systemscited by 25
  23. 2025Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributionscited by 12
  24. 2025MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sourcescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Membership Inference Attacks against Large Audio Language Modelscited by 2
  2. 2026Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scorescited by 2
  3. 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Taskscited by 1
  4. 2026The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detectioncited by 1
  5. 2026Provable Joint Decontamination for Benchmarking Multiple Large Language Modelscited by 0
  6. 2026Anchored or Drifting: What Recursive Self-Generation Reveals About Training Datacited by 0
  7. 2026PolERo: Studying Political Evasion in Romaniancited by 0
  8. 2026Structural Chain-of-Thought: Reasoning with Executable Computational Graphscited by 0