# Medical Imaging AI Competitions Lack Fairness

arXiv: https://arxiv.org/abs/2512.17581 · HTML: https://mlsanity.com/paper/2512.17581/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 19 Dec 2025.

## Builds on (most cited first, 15 of 15)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [Methods and open-source toolkit for analyzing and visualizing challenge results](https://mlsanity.com/paper/1910.05121/) (2019)
- [BIAS: Transparent reporting of biomedical image analysis challenges](https://mlsanity.com/paper/1910.04071/) (2019)
- [Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers](https://mlsanity.com/paper/2108.02922/) (2021)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [Copycats: the many lives of a publicly available medical imaging dataset](https://mlsanity.com/paper/2402.06353/) (2024)
- [False Promises in Medical Imaging AI? Assessing Validity of Outperformance Claims](https://mlsanity.com/paper/2505.04720/) (2025)
- [Climbing the Kaggle Leaderboard by Exploiting the Log-Loss Oracle](https://mlsanity.com/paper/1707.01825/) (2017)
- [Do Not Trust Licenses You See: Dataset Compliance Requires Massive-Scale AI-Powered Lifecycle Tracing](https://mlsanity.com/paper/2503.02784/) (2025)
- [Challenge Results Are Not Reproducible](https://mlsanity.com/paper/2307.07226/) (2023)
- [[Citation needed] Data usage and citation practices in medical imaging conferences](https://mlsanity.com/paper/2402.03003/) (2024)
- [A Technical Policy Blueprint for Trustworthy Decentralized AI](https://mlsanity.com/paper/2512.11878/) (2025)

## Built on by (most cited first, 1 of 1)

- [HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding](https://mlsanity.com/paper/2610.10156/) (2026)

## Abstract (arXiv)

Benchmarking competitions are central to the development of artificial intelligence (AI) in medical imaging, defining performance standards and shaping methodological progress. However, it remains unclear whether these benchmarks provide data that are sufficiently representative, accessible, and reusable to support clinically meaningful AI. In this work, we assess fairness along two complementary dimensions: (1) whether challenge datasets capture the diversity of real-world clinical data, and (2) whether they are accessible and legally reusable in line with the FAIR principles. To address this question, we conducted a large-scale systematic study of 249 biomedical image analysis challenges comprising 458 tasks across 19 imaging modalities. Our findings reveal limited representation across challenge datasets with respect to geographic location, imaging modalities, and problem types, raising concerns about how well current benchmarks reflect real-world clinical diversity. Despite their widespread influence, challenge datasets were frequently constrained by restrictive or ambiguous access conditions, inconsistent or non-compliant licensing practices, and incomplete documentation, limiting reproducibility and long-term reuse. Together, these shortcomings expose foundational fairness limitations in our benchmarking ecosystem and highlight a disconnect between leaderboard success and clinical relevance.