mlsanityData to 14 Oct 2026

We Need to Measure Data Diversity in NLP -- Better and Broader

arXiv 2505.20264 ↗First posted 26 May 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Although diversity in NLP datasets has received growing attention, the question of how to measure it remains largely underexplored. This opinion paper examines the conceptual and methodological challenges of measuring data diversity and argues that interdisciplinary perspectives are essential for developing more fine-grained and valid measures.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 47 of its 92 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

47 of 92 references.

References · 47

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2018Datasheets for Datasetscited by 1,386
  3. 2023LIMA: Less Is More for Alignmentcited by 1,100
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  6. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  7. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  8. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  9. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 294
  10. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  11. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 260
  12. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
35 more
  1. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  2. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  3. 2024A Survey on Data Selection for Language Modelscited by 236
  4. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  5. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  6. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
  7. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 130
  8. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 129
  9. 2023What's In My Big Data?cited by 129
  10. 2020BLEU might be Guilty but References are not Innocentcited by 110
  11. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  12. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 94
  13. 2022Same Author or Just Same Topic? Towards Content-Independent Style Representationscited by 86
  14. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  15. 2025Position: Evaluating Generative AI Systems Is a Social Science Measurement Challengecited by 63
  16. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 53
  17. 2021Changing the World by Changing the Datacited by 51
  18. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
  19. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 50
  20. 2020Predicting Performance for Natural Language Processing Taskscited by 49
  21. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasetscited by 36
  22. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  23. 2025Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metriccited by 24
  24. 2020Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collectionscited by 22
  25. 2022Measuring and Improving Semantic Diversity of Dialogue Generationcited by 16
  26. 2023Why Can't Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversitycited by 15
  27. 2024ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Lawscited by 15
  28. 2024The Sociolinguistic Foundations of Language Modelingcited by 13
  29. 2022What Makes Reading Comprehension Questions Difficult?cited by 11
  30. 2024A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Setscited by 10
  31. 2021Diversity-Aware Batch Active Learning for Dependency Parsingcited by 9
  32. 2024Racial/Ethnic Categories in AI and Algorithmic Fairness: Why They Matter and What They Representcited by 5
  33. 2022Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasetscited by 4
  34. 2024Variationist: Exploring Multifaceted Variation and Bias in Written Language Datacited by 3
  35. 2024PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematicscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026emb-diversity: A Tool for Embedding-Based Measurement of Data Diversitycited by 1
  2. 2026A Cross Community Agenda for Speech AIcited by 0