mlsanityData to 7 Oct 2026

Data Contamination Report from the 2024 CONDA Shared Task

arXiv 2407.21530 ↗First posted 31 Jul 2024cs.CL cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform. The platform continues to be online, open to contributions from the community.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 75 of its 99 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

75 of 99 references.

References · 75

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  13. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  14. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  15. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  16. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  17. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  18. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  19. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  20. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  21. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  22. 2018CoQA: A Conversational Question Answering Challengecited by 674
  23. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  24. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  25. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  26. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  27. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  28. 2023Llemma: An Open Language Model For Mathematicscited by 385
  29. 2018QuAC : Question Answering in Contextcited by 379
  30. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 361
  31. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  32. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  33. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  34. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  35. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  36. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  37. 2022Data Contamination: From Memorization to Exploitationcited by 234
  38. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 56
  2. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 49
  3. 2024Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?cited by 30
  4. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  5. 2024Detecting Training Data of Large Language Models via Expectation Maximizationcited by 13
  6. 2025Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translationcited by 13
  7. 2025Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Indexcited by 13
  8. 2024A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLIcited by 8
  9. 2025Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Modelscited by 8
  10. 2026The Impact of Post-training on Data Contaminationcited by 4
  11. 2025CoreEval: Automatically Building Contamination-Resilient Datasets with Real-World Knowledge toward Reliable LLM Evaluationcited by 3
  12. 2026When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluationcited by 3
7 more
  1. 2024Contamination Report for Multilingual Benchmarkscited by 2
  2. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  3. 2026Quantifying the Effect of Test Set Contamination on Generative Evaluationscited by 1
  4. 2026Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Misscited by 1
  5. 2025Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Networkcited by 0
  6. 2026Rethinking Easy-to-Hard: Limits of Curriculum Learning in Post-Training for Deductive Reasoningcited by 0
  7. 2026Evaluation-as-Search: Adaptive Discovery of Grounding Failures in Meeting Assistantscited by 0