# Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals

arXiv: https://arxiv.org/abs/2201.07040 · HTML: https://mlsanity.com/paper/2201.07040/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 18 Jan 2022.

## Builds on (most cited first, 17 of 17)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [emrQA: A Large Corpus for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1809.00732/) (2018)
- [Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research](https://mlsanity.com/paper/2112.01716/) (2021)
- [A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images](https://mlsanity.com/paper/2010.10563/) (2020)
- [An Empirical Evaluation of Deep Learning for ICD-9 Code Assignment using MIMIC-III Clinical Notes](https://mlsanity.com/paper/1802.02311/) (2018)
- [Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT](https://mlsanity.com/paper/2107.04374/) (2021)
- [A curated, ontology-based, large-scale knowledge graph of artificial intelligence tasks and benchmarks](https://mlsanity.com/paper/2110.01434/) (2021)
- [SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search](https://mlsanity.com/paper/2010.05987/) (2020)
- [Classification of Radiology Reports Using Neural Attention Models](https://mlsanity.com/paper/1708.06828/) (2017)
- [Towards better healthcare: What could and should be automated?](https://mlsanity.com/paper/1910.09444/) (2019)
- [FindZebra: A search engine for rare diseases](https://mlsanity.com/paper/1303.3229/) (2013)
- [Repurposing TREC-COVID Annotations to Answer the Key Questions of CORD-19](https://mlsanity.com/paper/2008.12353/) (2020)
- [Benchmarking Clinical Decision Support Search](https://mlsanity.com/paper/1801.09322/) (2018)

## Built on by (most cited first, 16 of 16)

- [Mapping global dynamics of benchmark creation and saturation in artificial intelligence](https://mlsanity.com/paper/2203.04592/) (2022)
- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs](https://mlsanity.com/paper/2303.12961/) (2023)
- [Computational Hermeneutics: Evaluating generative AI as a cultural technology](https://mlsanity.com/paper/2604.16403/) (2026)
- [BenchMD: A Benchmark for Unified Learning on Medical Images and Sensors](https://mlsanity.com/paper/2304.08486/) (2023)
- [CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews](https://mlsanity.com/paper/2311.12474/) (2023)
- [Towards Experience-Centered AI: A Framework for Integrating Lived Experience in Design and Development](https://mlsanity.com/paper/2508.06849/) (2025)
- [PhilHumans: Benchmarking Machine Learning for Personal Health](https://mlsanity.com/paper/2405.02770/) (2024)
- [Enhanced prediction of spine surgery outcomes using advanced machine learning techniques and oversampling methods](https://mlsanity.com/paper/2503.18996/) (2025)
- [Deprecating Benchmarks: Criteria and Framework](https://mlsanity.com/paper/2507.06434/) (2025)
- [Performance Assessment Strategies for Language Model Applications in Healthcare](https://mlsanity.com/paper/2509.08087/) (2025)
- [Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes](https://mlsanity.com/paper/2511.20680/) (2025)
- [Which Anatomy Matters Under Limited Labels? A Data-Efficient Anatomy-Aware Benchmark for Cardiac Pathology Prediction](https://mlsanity.com/paper/2606.06509/) (2026)
- [Position: Medical AI Neglects Real Treatment Outcomes](https://mlsanity.com/paper/2608.14598/) (2026)
- [SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence](https://mlsanity.com/paper/2608.29582/) (2026)
- [Structural Alignment for Reliable Industrial AI: Bridging Physical Reality, Data, Models, and Human Intent](https://mlsanity.com/paper/2609.35400/) (2026)

## Abstract (arXiv)

Publicly accessible benchmarks that allow for assessing and comparing model performances are important drivers of progress in artificial intelligence (AI). While recent advances in AI capabilities hold the potential to transform medical practice by assisting and augmenting the cognitive processes of healthcare professionals, the coverage of clinically relevant tasks by AI benchmarks is largely unclear. Furthermore, there is a lack of systematized meta-information that allows clinical AI researchers to quickly determine accessibility, scope, content and other characteristics of datasets and benchmark datasets relevant to the clinical domain. To address these issues, we curated and released a comprehensive catalogue of datasets and benchmarks pertaining to the broad domain of clinical and biomedical natural language processing (NLP), based on a systematic review of literature and online resources. A total of 450 NLP datasets were manually systematized and annotated with rich metadata, such as targeted tasks, clinical applicability, data types, performance metrics, accessibility and licensing information, and availability of data splits. We then compared tasks covered by AI benchmark datasets with relevant tasks that medical practitioners reported as highly desirable targets for automation in a previous empirical study. Our analysis indicates that AI benchmarks of direct clinical relevance are scarce and fail to cover most work activities that clinicians want to see addressed. In particular, tasks associated with routine documentation and patient data administration workflows are not represented despite significant associated workloads. Thus, currently available AI benchmarks are improperly aligned with desired targets for AI automation in clinical settings, and novel benchmarks should be created to fill these gaps.