mlsanityData to 14 Oct 2026

Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs

arXiv 2402.03927 ↗First posted 6 Feb 2024cs.CL cs.AIcited by 229

Not on the map. Why?

Cited by 23 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Natural Language Processing (NLP) research is increasingly focusing on the use of Large Language Models (LLMs), with some of the most popular ones being either fully or partially closed-source. The lack of access to model details, especially regarding training data, has repeatedly raised concerns about data contamination among researchers. Several attempts have been made to address this issue, but they are limited to anecdotal evidence and trial and error. Additionally, they overlook the problem of indirect data leaking, where models are iteratively improved by using data coming from users. In this work, we conduct the first systematic analysis of work using OpenAI's GPT-3.5 and GPT-4, the most prominently used LLMs today, in the context of data contamination. By analysing 255 papers and considering OpenAI's data usage policy, we extensively document the amount of data leaked to these models during the first year after the model's release. We report that these models have been globally exposed to ~4.7M samples from 263 benchmarks. At the same time, we document a number of evaluation malpractices emerging in the reviewed papers, such as unfair or missing baseline comparisons and reproducibility issues. We release our results as a collaborative project on https://leak-llm.github.io/, where other researchers can contribute to our efforts.

Key facts. Cited by 229 papers in the map (data through 14 Oct 2026). Builds on 155 of its 248 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

155 of 248 references.

References · 155

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
38 more
  1. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  2. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  3. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  4. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  5. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  6. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  7. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  8. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  9. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  10. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  11. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  12. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  13. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  14. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  15. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  16. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
  17. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  18. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 343
  19. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  20. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  21. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  22. 2023How is ChatGPT's behavior changing over time?cited by 270
  23. 2023Mathematical Capabilities of ChatGPTcited by 268
  24. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  25. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  26. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 246
  27. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 238
  28. 2022Data Contamination: From Memorization to Exploitationcited by 236
  29. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  30. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  31. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  32. 2022Is GPT-3 a Good Data Annotator?cited by 208
  33. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
  34. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  35. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  36. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  37. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  38. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172

50 most cited of 155 All 155 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 71; 2025: 93; 2026: 65.

229 papers.

Sort:

By area:

  1. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  2. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  3. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  4. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  5. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  6. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  7. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 110
  8. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 85
  9. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 83
  10. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  11. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  12. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
38 more
  1. 2024VariErr NLI: Separating Annotation Error from Human Label Variationcited by 42
  2. 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 40
  3. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  4. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  5. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 37
  6. 2025LongCodeBench: Evaluating Coding LLMs at 1M Context Windowscited by 37
  7. 2025A Survey on Data Contamination for Large Language Modelscited by 35
  8. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  9. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  10. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  11. 2024Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?cited by 32
  12. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  13. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languagescited by 28
  14. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  15. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  16. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  17. 2025Ontology Generation using Large Language Modelscited by 25
  18. 2024TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMscited by 24
  19. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  20. 2024Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmarkcited by 23
  21. 2024LLM Uncertainty Quantification through Directional Entailment Graph and Claim Level Response Augmentationcited by 23
  22. 2024The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?cited by 22
  23. 2024A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Changecited by 21
  24. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Datacited by 21
  25. 2024Data Contamination Report from the 2024 CONDA Shared Taskcited by 20
  26. 2025Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboardscited by 20
  27. 2024Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challengescited by 18
  28. 2024Black-box Uncertainty Quantification Method for LLM-as-a-Judgecited by 18
  29. 2024The Open Source Advantage in Large Language Models (LLMs)cited by 18
  30. 2025LLM-Safety Evaluations Lack Robustnesscited by 18
  31. 2025Evaluating LLM Metrics Through Real-World Capabilitiescited by 18
  32. 2024SEC-QA: A Systematic Evaluation Corpus for Financial QAcited by 17
  33. 2024"Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?cited by 17
  34. 2025How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergencecited by 15
  35. 2024Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedbackcited by 14
  36. 2025LastingBench: Defend Benchmarks Against Knowledge Leakagecited by 14
  37. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  38. 2024The Sociolinguistic Foundations of Language Modelingcited by 13

50 most cited of 229 All 229 (JSON)