mlsanityData to 13 Oct 2026

Data and its (dis)contents: A survey of dataset development and use in machine learning research

arXiv 2012.05345 ↗First posted 9 Dec 2020cs.LGcited by 259

Not on the map. Why?

Abstract (arXiv)

Datasets have played a foundational role in the advancement of machine learning research. They form the basis for the models we design and deploy, as well as our primary medium for benchmarking and evaluation. Furthermore, the ways in which we collect, construct and share these datasets inform the kinds of problems the field pursues and the methods explored in algorithm development. However, recent work from a breadth of perspectives has revealed the limitations of predominant practices in dataset collection and use. In this paper, we survey the many concerns raised about the way we collect and use data in machine learning and advocate that a more cautious and thorough understanding of data is necessary to address several of the practical and ethical issues of the field.

Key facts. Cited by 259 papers in the map (data through 13 Oct 2026). First posted on arXiv 9 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 34; 2022: 61; 2023: 56; 2024: 44; 2025: 37; 2026: 27.

259 papers.

Sort:

By area:

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
  2. 2023PaLM 2 Technical Reportcited by 1,162
  3. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 641
  4. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 485
  5. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 444
  6. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 428
  7. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 344
  8. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 331
  9. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  10. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  11. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  12. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
38 more
  1. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 178
  2. 2021A survey on datasets for fairness-aware machine learningcited by 157
  3. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  4. 2023Auditing large language models: a three-layered approachcited by 136
  5. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 129
  6. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  7. 20223D Common Corruptions and Data Augmentationcited by 119
  8. 2021Gender Bias in Machine Translationcited by 116
  9. 2023Foundation Models and Fair Usecited by 116
  10. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  11. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 103
  12. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 101
  13. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  14. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  15. 2024Better than classical? The subtle art of benchmarking quantum machine learning modelscited by 85
  16. 2021A Study of Face Obfuscation in ImageNetcited by 83
  17. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  18. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  19. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 75
  20. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  21. 2022Leakage and the Reproducibility Crisis in ML-based Sciencecited by 65
  22. 2025Authenticated Delegation and Authorized AI Agentscited by 63
  23. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 61
  24. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 59
  25. 2021OpenFL: An open-source framework for Federated Learningcited by 58
  26. 2023FACET: Fairness in Computer Vision Evaluation Benchmarkcited by 58
  27. 2023Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generatorscited by 56
  28. 2023AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asiacited by 54
  29. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  30. 2022Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contextscited by 53
  31. 2021Changing the World by Changing the Datacited by 51
  32. 2021PASS: An ImageNet replacement for self-supervised pretraining without humanscited by 51
  33. 2021Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Paperscited by 49
  34. 2022FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processingcited by 48
  35. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  36. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45
  37. 2022Handling and Presenting Harmful Text in NLP Researchcited by 43
  38. 2022Contrastive Language-Vision AI Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Biascited by 43

50 most cited of 259 All 259 (JSON)