# Dataset Diversity Metrics and Impact on Classification Models

arXiv: https://arxiv.org/abs/2603.15276 · HTML: https://mlsanity.com/paper/2603.15276/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity](https://mlsanity.com/paper/2510.01171/) (2025)
- [Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models](https://mlsanity.com/paper/2404.07983/) (2024)
- [Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning](https://mlsanity.com/paper/1809.10780/) (2018)
- [Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI](https://mlsanity.com/paper/2409.14160/) (2024)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [Image Generation Diversity Issues and How to Tame Them](https://mlsanity.com/paper/2411.16171/) (2024)
- [Detecting Shortcuts in Medical Images -- A Case Study in Chest X-rays](https://mlsanity.com/paper/2211.04279/) (2022)
- [Fr'echet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets](https://mlsanity.com/paper/2412.01496/) (2024)
- [Measuring Data](https://mlsanity.com/paper/2212.05129/) (2022)
- [When More is Less: Incorporating Additional Datasets Can Hurt Performance By Introducing Spurious Correlations](https://mlsanity.com/paper/2308.04431/) (2023)
- [Data Representativity for Machine Learning and AI Systems](https://mlsanity.com/paper/2203.04706/) (2022)
- [Measuring Diversity in Co-creative Image Generation](https://mlsanity.com/paper/2403.13826/) (2024)
- [The Data Addition Dilemma](https://mlsanity.com/paper/2408.04154/) (2024)
- [Position: All Current Generative Fidelity and Diversity Metrics are Flawed](https://mlsanity.com/paper/2505.22450/) (2025)
- [How You Split Matters: Data Leakage and Subject Characteristics Studies in Longitudinal Brain MRI Analysis](https://mlsanity.com/paper/2309.00350/) (2023)
- [In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review](https://mlsanity.com/paper/2501.10727/) (2025)
- [Measuring Diversity: Axioms and Challenges](https://mlsanity.com/paper/2410.14556/) (2024)
- [Mask of truth: model sensitivity to unexpected regions of medical images](https://mlsanity.com/paper/2412.04030/) (2024)

## Built on by (most cited first, 1 of 1)

- [Look What the Probes Dragged In! Real-World Chest X-ray Shortcuts in MedCLIP](https://mlsanity.com/paper/2608.12086/) (2026)

## Abstract (arXiv)

The diversity of training datasets is usually perceived as an important aspect to obtain a robust model. However, the definition of diversity is often not defined or differs across papers, and while some metrics exist, the quantification of this diversity is often overlooked when developing new algorithms. In this work, we study the behaviour of multiple dataset diversity metrics for image, text and metadata using MorphoMNIST, a toy dataset with controlled perturbations, and PadChest, a publicly available chest X-ray dataset. We evaluate whether these metrics correlate with each other but also with the intuition of a clinical expert. We also assess whether they correlate with downstream-task performance and how they impact the training dynamic of the models. We find limited correlations between the AUC and image or metadata reference-free diversity metrics, but higher correlations with the FID and the semantic diversity metrics. Finally, the clinical expert indicates that scanners are the main source of diversity in practice. However, we find that the addition of another scanner to the training set leads to shortcut learning. The code used in this study is available at https://github.com/TheoSourget/dataset_diversity_evaluation