mlsanityData to 7 Oct 2026

Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research

arXiv 2112.01716 ↗First posted 3 Dec 2021cs.LG cs.CL cs.CVcited by 100

Not on the map. Why?

Abstract (arXiv)

Benchmark datasets play a central role in the organization of machine learning research. They coordinate researchers around shared research problems and serve as a measure of progress towards shared goals. Despite the foundational role of benchmarking practices in this field, relatively little attention has been paid to the dynamics of benchmark dataset use and reuse, within or across machine learning subcommunities. In this paper, we dig into these dynamics. We study how dataset usage patterns differ across machine learning subcommunities and across time from 2015-2020. We find increasing concentration on fewer and fewer datasets within task communities, significant adoption of datasets from other tasks, and concentration across the field on datasets that have been introduced by researchers situated within a small number of elite institutions. Our results have implications for scientific evaluation, AI ethics, and equity/access within the field.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 19 of its 53 references that are in the map. First posted on arXiv 3 Dec 2021.

Builds on

19 of 53 references.

References · 19

  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  6. 2016MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognitioncited by 621
  7. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  8. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  9. 2020Are we done with ImageNet?cited by 329
  10. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  11. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  12. 2015The MegaFace Benchmark: 1 Million Faces for Recognition at Scalecited by 199
7 more
  1. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  2. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 152
  3. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  4. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  5. 2019Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?cited by 64
  6. 2020Targeting the Benchmark: On Methodology in Current Natural Language Processing Researchcited by 44
  7. 2019Value-laden Disciplinary Shifts in Machine Learningcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 26; 2023: 21; 2024: 27; 2025: 15; 2026: 11.

100 papers.

Sort:

By area:

  1. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 195
  2. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  3. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  4. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  5. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  6. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  7. 2024Better than classical? The subtle art of benchmarking quantum machine learning modelscited by 82
  8. 2022Mapping global dynamics of benchmark creation and saturation in artificial intelligencecited by 71
  9. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  10. 2023FACET: Fairness in Computer Vision Evaluation Benchmarkcited by 58
  11. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  12. 2022Evaluation Gaps in Machine Learning Practicecited by 48
38 more
  1. 2024A Decade's Battle on Dataset Bias: Are We There Yet?cited by 46
  2. 2022Handling and Presenting Harmful Text in NLP Researchcited by 43
  3. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  4. 2022Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluationcited by 39
  5. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  6. 2023Ecosystem Graphs: The Social Footprint of Foundation Modelscited by 31
  7. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  8. 2024Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on Hugging Facecited by 27
  9. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 26
  10. 2022Bugs in the Data: How ImageNet Misrepresents Biodiversitycited by 24
  11. 2023Learning from data with structured missingnesscited by 24
  12. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  13. 2022Fair ranking: a critical review, challenges, and future directionscited by 21
  14. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  15. 2023Ethical Considerations for Responsible Data Curationcited by 17
  16. 2022Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionalscited by 16
  17. 2023Aligning benchmark datasets for table structure recognitioncited by 15
  18. 2023A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why?cited by 14
  19. 2023Inferring Capabilities from Task Performance with Bayesian Triangulationcited by 13
  20. 2024From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolutioncited by 12
  21. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 12
  22. 2022Metaethical Perspectives on 'Benchmarking' AI Ethicscited by 11
  23. 2022The Algorithmic Imprintcited by 10
  24. 2024Benchmark Data Repositories for Better Benchmarkingcited by 10
  25. 2024Rethinking model prototyping through the MedMNIST+ dataset collectioncited by 9
  26. 2025What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Textcited by 9
  27. 2026No One Knows the State of the Art in Geospatial Foundation Modelscited by 9
  28. 2024Pricing and Competition for Generative AIcited by 8
  29. 2024What Do Machine Learning Researchers Mean by "Reproducible"?cited by 8
  30. 2025How Reliable is Language Model Micro-Benchmarking?cited by 8
  31. 2022Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASRcited by 7
  32. 2022On the role of benchmarking data sets and simulations in method comparison studiescited by 7
  33. 2022HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictionscited by 6
  34. 2023A View From Somewhere: Human-Centric Face Representationscited by 6
  35. 2024Oil & Water? Diffusion of AI Within and Across Scientific Fieldscited by 6
  36. 2024Copycats: the many lives of a publicly available medical imaging datasetcited by 5
  37. 2024Building Better Datasets: Seven Recommendations for Responsible Design from Dataset Creatorscited by 5
  38. 2022Visual Ground Truth Construction as Faceted Classificationcited by 4

50 most cited of 100 All 100 (JSON)