mlsanityData to 14 Oct 2026

Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation

arXiv 2309.10677 ↗First posted 19 Sep 2023cs.CL cs.AIcited by 48

Not on the map. Why?

Abstract (arXiv)

Data contamination in model evaluation is getting increasingly prevalent as the massive training corpora of large language models often unintentionally include benchmark samples. Therefore, contamination analysis has became an inevitable part of reliable model evaluation. However, existing method of contamination analysis requires the access of the entire training data which is often confidential for recent models. This prevent the community to rigorously audit these models and conduct accurate assessment of their capability. In this paper, we propose a novel method to quantify contamination without the access of the full training set, that measure the extent of contamination with perplexity. Our analysis provides evidence of significant memorisation of recent foundation models in popular reading comprehension, summarisation benchmarks, while multiple choice appears less contaminated.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 14 of its 17 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

14 of 17 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  12. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
2 more
  1. 2018QuAC : Question Answering in Contextcited by 381
  2. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 17; 2025: 15; 2026: 12.

48 papers.

Sort:

By area:

  1. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  2. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  3. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 106
  4. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 99
  5. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  6. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 59
  7. 2023LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Constructioncited by 58
  8. 2023An Open Source Data Contamination Report for Large Language Modelscited by 51
  9. 2025The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reasoncited by 49
  10. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 48
  11. 2025A Survey on Data Contamination for Large Language Modelscited by 35
  12. 2024Evading Data Contamination Detection for Language Models is (too) Easycited by 30
36 more
  1. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  2. 2024Data Contamination Can Cross Language Barrierscited by 26
  3. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  4. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  5. 2024Synthesizing Post-Training Data for LLMs through Multi-Agent Simulationcited by 24
  6. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  7. 2025How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergencecited by 15
  8. 2025Forget What You Know about LLMs Evaluations -- LLMs are Like a Chameleoncited by 13
  9. 2024A Comprehensive Survey of Contamination Detection Methods in Large Language Modelscited by 12
  10. 2024Are Large Language Models Memorizing Bug Benchmarks?cited by 12
  11. 2025DCR: Quantifying Data Contamination in LLMs Evaluationcited by 10
  12. 2025NetArena: Dynamic Benchmarks for AI Agents in Network Automationcited by 9
  13. 2024Fine-tuning can Help Detect Pretraining Data from Large Language Modelscited by 8
  14. 2024PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Modelscited by 6
  15. 2025Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Modelscited by 6
  16. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  17. 2024Evaluating Large Language Models for Generalization and Robustness via Data Compressioncited by 5
  18. 2025RefineStat: Efficient Exploration for Probabilistic Program Synthesiscited by 5
  19. 2025Detecting Distillation Data from Reasoning Modelscited by 4
  20. 2025Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysiscited by 3
  21. 2025Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluationcited by 2
  22. 2025LogitTrace: Detecting Benchmark Contamination via Layerwise Logit Trajectoriescited by 2
  23. 2026Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysiscited by 2
  24. 2025AdEval: Alignment-based Dynamic Evaluation to Mitigate Data Contamination in Large Language Modelscited by 1
  25. 2025Provable Training Data Identification for Large Language Modelscited by 1
  26. 2026Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implicationscited by 1
  27. 2025LNE-Blocking: An Efficient Framework for Contamination Mitigation Evaluation on Large Language Modelscited by 0
  28. 2026No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Modelscited by 0
  29. 2026From Unfamiliar to Familiar: Detecting Pre-training Data via Gradient Deviations in Large Language Modelscited by 0
  30. 2026When AI Navigates the Fog of Warcited by 0
  31. 2026A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repaircited by 0
  32. 2026Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneitycited by 0
  33. 2026An Interpretable and Scalable Framework for Evaluating Large Language Modelscited by 0
  34. 2026Predicting Task Difficulty Without Rolloutscited by 0
  35. 2026Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contaminationcited by 0
  36. 2026WildTrace: Evaluating Training Data Detection in Realistic Large-Scale LLM Post-Trainingcited by 0