mlsanityData to 7 Oct 2026

Position: Measure Dataset Diversity, Don't Just Claim It

arXiv 2407.08188 ↗First posted 11 Jul 2024cs.LG cs.CYcited by 26

Not on the map. Why?

Abstract (arXiv)

Machine learning (ML) datasets, often perceived as neutral, inherently encapsulate abstract and disputed social constructs. Dataset curators frequently employ value-laden terms such as diversity, bias, and quality to characterize datasets. Despite their prevalence, these terms lack clear definitions and validation. Our research explores the implications of this issue by analyzing "diversity" across 135 image and text datasets. Drawing from social sciences, we apply principles from measurement theory to identify considerations and offer recommendations for conceptualizing, operationalizing, and evaluating diversity in datasets. Our findings have broader implications for ML research, advocating for a more nuanced and precise approach to handling value-laden properties in dataset construction.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 140 of its 231 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

140 of 231 references.

References · 140

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Segment Anythingcited by 8,158
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  11. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  12. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
38 more
  1. 2018Datasheets for Datasetscited by 1,371
  2. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  6. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  7. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  8. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  9. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  10. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  11. 2018Gender Bias in Coreference Resolutioncited by 438
  12. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  13. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  14. 2018TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wildcited by 342
  15. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  16. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  17. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  18. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  19. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  20. 2019Measurement and Fairnesscited by 299
  21. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  22. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  23. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  24. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  25. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  26. 2019Human uncertainty makes classification more robustcited by 231
  27. 2019Connecting Vision and Language with Localized Narrativescited by 225
  28. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  29. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  30. 2021Fake It Till You Make It: Face analysis in the wild using synthetic data alonecited by 200
  31. 2020Deblurring by Realistic Blurringcited by 187
  32. 2018Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowdscited by 186
  33. 2021FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Datasetcited by 179
  34. 2021The Values Encoded in Machine Learning Researchcited by 172
  35. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  36. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  37. 2021FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured informationcited by 166
  38. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 152

50 most cited of 140 All 140 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 7.

26 papers.

Sort:

By area:

  1. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  2. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  3. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  4. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  5. 2024Image Generation Diversity Issues and How to Tame Themcited by 22
  6. 2026Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teamingcited by 15
  7. 2025Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparencycited by 14
  8. 2025Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencodercited by 11
  9. 2024Measuring Diversity: Axioms and Challengescited by 6
  10. 2026Limited Linguistic Diversity in Embodied AI Datasetscited by 5
  11. 2025Position: Beyond Assistance -- Reimagining LLMs as Ethical and Adaptive Co-Creators in Mental Health Carecited by 3
  12. 2025Ontologies in Design: How Imagining a Tree Reveals Possibilites and Assumptions in Large Language Modelscited by 3
14 more
  1. 2025From Macro to Micro: Probing Dataset Diversity in Language Model Fine-Tuningcited by 3
  2. 2025Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systemscited by 3
  3. 2025We Need to Measure Data Diversity in NLP -- Better and Broadercited by 2
  4. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  5. 2025We Need Improved Data Curation and Attribution in AI for Scientific Discoverycited by 1
  6. 2026Dataset Diversity Metrics and Impact on Classification Modelscited by 1
  7. 2026Knowledge Distillation Must Account for What It Losescited by 1
  8. 2026emb-diversity: A Tool for Embedding-Based Measurement of Data Diversitycited by 1
  9. 2024Saliency-Based diversity and fairness Metric and FaceKeepOriginalAugment: A Novel Approach for Enhancing Fairness and Diversitycited by 0
  10. 2025TEDI: Trustworthy and Ethical Dataset Indicators to Analyze and Compare Dataset Documentationcited by 0
  11. 2025Intuitions of Machine Learning Researchers about Transfer Learning for Medical Image Classificationcited by 0
  12. 2025DiRe: Diversity-promoting Regularization for Dataset Condensationcited by 0
  13. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0
  14. 2026Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluationcited by 0