# Data and its (dis)contents: A survey of dataset development and use in machine learning research

arXiv: https://arxiv.org/abs/2012.05345 · HTML: https://mlsanity.com/paper/2012.05345/

## Key facts

- Cited by 260 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 30 of 260)

- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale](https://mlsanity.com/paper/2211.03759/) (2022)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets](https://mlsanity.com/paper/2106.10328/) (2021)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [A survey on datasets for fairness-aware machine learning](https://mlsanity.com/paper/2110.00530/) (2021)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [3D Common Corruptions and Data Augmentation](https://mlsanity.com/paper/2203.01441/) (2022)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [Foundation Models and Fair Use](https://mlsanity.com/paper/2303.15715/) (2023)
- [Understanding and Evaluating Racial Biases in Image Captioning](https://mlsanity.com/paper/2106.08503/) (2021)
- [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://mlsanity.com/paper/2306.07899/) (2023)
- [Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research](https://mlsanity.com/paper/2112.01716/) (2021)
- [DataPerf: Benchmarks for Data-Centric AI Development](https://mlsanity.com/paper/2207.10062/) (2022)
- [Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development](https://mlsanity.com/paper/2108.04308/) (2021)
- [Better than classical? The subtle art of benchmarking quantum machine learning models](https://mlsanity.com/paper/2403.07059/) (2024)
- [A Study of Face Obfuscation in ImageNet](https://mlsanity.com/paper/2103.06191/) (2021)
- [What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus](https://mlsanity.com/paper/2105.02732/) (2021)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)

## Abstract (arXiv)

Datasets have played a foundational role in the advancement of machine learning research. They form the basis for the models we design and deploy, as well as our primary medium for benchmarking and evaluation. Furthermore, the ways in which we collect, construct and share these datasets inform the kinds of problems the field pursues and the methods explored in algorithm development. However, recent work from a breadth of perspectives has revealed the limitations of predominant practices in dataset collection and use. In this paper, we survey the many concerns raised about the way we collect and use data in machine learning and advocate that a more cautious and thorough understanding of data is necessary to address several of the practical and ethical issues of the field.