mlsanityData to 7 Oct 2026

Measuring Data

arXiv 2212.05129 ↗First posted 9 Dec 2022cs.AI cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

We identify the task of measuring data to quantitatively characterize the composition of machine learning data and datasets. Similar to an object's height, width, and volume, data measurements quantify different attributes of data along common dimensions that support comparison. Several lines of research have proposed what we refer to as measurements, with differing terminology; we bring some of this work together, particularly in fields of computer vision and language, and build from it to motivate measuring data as a critical component of responsible AI development. Measuring data aids in systematically building and analyzing machine learning (ML) data towards specific goals and gaining better control of what modern ML systems will learn. We conclude with a discussion of the many avenues of future work, the limitations of data measurements, and how to leverage these measurement approaches in research and practice.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 35 of its 108 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

35 of 108 references.

References · 35

  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  2. 2014Going Deeper with Convolutionscited by 4,231
  3. 2016Improved Techniques for Training GANscited by 2,938
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2018Datasheets for Datasetscited by 1,371
  6. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  7. 2018Demystifying MMD GANscited by 1,268
  8. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  11. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  12. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
23 more
  1. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  2. 2019Measurement and Fairnesscited by 299
  3. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  4. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  5. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  6. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  7. 2019Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learningcited by 116
  8. 2021Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspectivecited by 116
  9. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  10. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  11. 2021Language Model Evaluation Beyond Perplexitycited by 77
  12. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 52
  13. 2022BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Samplingcited by 30
  14. 2021A Data-Centric Approach for Training Deep Neural Networks with Less Datacited by 28
  15. 2020Diversity and Inclusion Metrics in Subset Selectioncited by 27
  16. 2021Measuring Model Biases in the Absence of Ground Truthcited by 24
  17. 2022Bugs in the Data: How ImageNet Misrepresents Biodiversitycited by 24
  18. 2020Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collectionscited by 21
  19. 2022SEAL : Interactive Tool for Systematic Error Analysis and Labelingcited by 15
  20. 2015A Survey of Current Datasets for Vision and Language Researchcited by 13
  21. 2021SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarizationcited by 13
  22. 2017Outlier Detection for Text Data : An Extended Versioncited by 8
  23. 2022DataLab: A Platform for Data Analysis and Interventioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 9; 2025: 1; 2026: 1.

18 papers.

Sort:

By area:

  1. 2023StarCoder: may the source be with you!cited by 790
  2. 2023What's In My Big Data?cited by 129
  3. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  4. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  5. 2024Foundation Models for Music: A Surveycited by 34
  6. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 30
  7. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languagescited by 28
  8. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  9. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
  10. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  11. 2024The Data Addition Dilemmacited by 15
  12. 2024Training Compute Thresholds: Features and Functions in AI Regulationcited by 13
6 more
  1. 2023Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Developmentcited by 6
  2. 2024A Survey of Transformer-based Language Models with Focus on Efficiencycited by 6
  3. 2024Data Measurements for Decentralized Data Marketscited by 2
  4. 2026Dataset Diversity Metrics and Impact on Classification Modelscited by 1
  5. 2024Standardness Clouds Meaning: A Position Regarding the Informed Usage of Standard Datasetscited by 0
  6. 2025TEDI: Trustworthy and Ethical Dataset Indicators to Analyze and Compare Dataset Documentationcited by 0