# Measuring Data

arXiv: https://arxiv.org/abs/2212.05129 · HTML: https://mlsanity.com/paper/2212.05129/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 108 references that are in the map.
- First posted on arXiv 9 Dec 2022.

## Builds on (most cited first, 30 of 35)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning](https://mlsanity.com/paper/1912.10389/) (2019)
- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [On the State of the Art of Evaluation in Neural Language Models](https://mlsanity.com/paper/1707.05589/) (2017)
- [Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development](https://mlsanity.com/paper/2108.04308/) (2021)
- [Language Model Evaluation Beyond Perplexity](https://mlsanity.com/paper/2106.00085/) (2021)
- [Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection](https://mlsanity.com/paper/2201.10474/) (2022)
- [BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling](https://mlsanity.com/paper/2207.06814/) (2022)
- [A Data-Centric Approach for Training Deep Neural Networks with Less Data](https://mlsanity.com/paper/2110.03613/) (2021)
- [Diversity and Inclusion Metrics in Subset Selection](https://mlsanity.com/paper/2002.03256/) (2020)
- [Measuring Model Biases in the Absence of Ground Truth](https://mlsanity.com/paper/2103.03417/) (2021)
- [Bugs in the Data: How ImageNet Misrepresents Biodiversity](https://mlsanity.com/paper/2208.11695/) (2022)
- [Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections](https://mlsanity.com/paper/2003.08529/) (2020)

## Built on by (most cited first, 18 of 18)

- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [What's In My Big Data?](https://mlsanity.com/paper/2310.20707/) (2023)
- [Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining](https://mlsanity.com/paper/2212.06470/) (2022)
- [MathPile: A Billion-Token-Scale Pretraining Corpus for Math](https://mlsanity.com/paper/2312.17120/) (2023)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [The ROOTS Search Tool: Data Transparency for LLMs](https://mlsanity.com/paper/2302.14035/) (2023)
- [LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages](https://mlsanity.com/paper/2406.06196/) (2024)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models](https://mlsanity.com/paper/2308.16458/) (2023)
- [On Catastrophic Inheritance of Large Foundation Models](https://mlsanity.com/paper/2402.01909/) (2024)
- [The Data Addition Dilemma](https://mlsanity.com/paper/2408.04154/) (2024)
- [Training Compute Thresholds: Features and Functions in AI Regulation](https://mlsanity.com/paper/2405.10799/) (2024)
- [Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Development](https://mlsanity.com/paper/2307.11525/) (2023)
- [A Survey of Transformer-based Language Models with Focus on Efficiency](https://mlsanity.com/paper/2406.16893/) (2024)
- [Data Measurements for Decentralized Data Markets](https://mlsanity.com/paper/2406.04257/) (2024)
- [Dataset Diversity Metrics and Impact on Classification Models](https://mlsanity.com/paper/2603.15276/) (2026)
- [Standardness Clouds Meaning: A Position Regarding the Informed Usage of Standard Datasets](https://mlsanity.com/paper/2406.13552/) (2024)
- [TEDI: Trustworthy and Ethical Dataset Indicators to Analyze and Compare Dataset Documentation](https://mlsanity.com/paper/2505.17841/) (2025)

## Abstract (arXiv)

We identify the task of measuring data to quantitatively characterize the composition of machine learning data and datasets. Similar to an object's height, width, and volume, data measurements quantify different attributes of data along common dimensions that support comparison. Several lines of research have proposed what we refer to as measurements, with differing terminology; we bring some of this work together, particularly in fields of computer vision and language, and build from it to motivate measuring data as a critical component of responsible AI development. Measuring data aids in systematically building and analyzing machine learning (ML) data towards specific goals and gaining better control of what modern ML systems will learn. We conclude with a discussion of the many avenues of future work, the limitations of data measurements, and how to leverage these measurement approaches in research and practice.