# MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation

arXiv: https://arxiv.org/abs/2110.01406 · HTML: https://mlsanity.com/paper/2110.01406/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 143 references that are in the map.
- First posted on arXiv 29 Sep 2021.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning](https://mlsanity.com/paper/1702.07464/) (2017)
- [CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation](https://mlsanity.com/paper/2001.06535/) (2020)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation](https://mlsanity.com/paper/1810.04304/) (2018)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [Common Limitations of Image Processing Metrics: A Picture Story](https://mlsanity.com/paper/2104.05642/) (2021)
- [OpenFL: An open-source framework for Federated Learning](https://mlsanity.com/paper/2105.06413/) (2021)
- [The Federated Tumor Segmentation (FeTS) Challenge](https://mlsanity.com/paper/2105.05874/) (2021)
- [GaNDLF: A Generally Nuanced Deep Learning Framework for Scalable End-to-End Clinical Workflows in Medical Imaging](https://mlsanity.com/paper/2103.01006/) (2021)
- [Spark NLP: Natural Language Understanding at Scale](https://mlsanity.com/paper/2101.10848/) (2021)
- [Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs](https://mlsanity.com/paper/2008.07180/) (2020)

## Built on by (most cited first, 30 of 88)

- [MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine](https://mlsanity.com/paper/2408.02900/) (2024)
- [WDM: 3D Wavelet Diffusion Models for High-Resolution Medical Image Synthesis](https://mlsanity.com/paper/2402.19043/) (2024)
- [Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2404.06448/) (2024)
- [nnInteractive: Redefining 3D Promptable Segmentation](https://mlsanity.com/paper/2503.08373/) (2025)
- [Denoising Diffusion Models for 3D Healthy Brain Tissue Inpainting](https://mlsanity.com/paper/2403.14499/) (2024)
- [Analysis of the BraTS 2023 Intracranial Meningioma Segmentation Challenge](https://mlsanity.com/paper/2405.09787/) (2024)
- [Denoising Diffusion Models for Inpainting of Healthy Brain Tissue](https://mlsanity.com/paper/2402.17307/) (2024)
- [Advanced Tumor Segmentation in Medical Imaging: An Ensemble Approach for BraTS 2023 Adult Glioma and Pediatric Tumor Tasks](https://mlsanity.com/paper/2403.09262/) (2024)
- [LesionLocator: Zero-Shot Universal Tumor Segmentation and Tracking in 3D Whole-Body Imaging](https://mlsanity.com/paper/2502.20985/) (2025)
- [BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text](https://mlsanity.com/paper/2504.19467/) (2025)
- [BraTS-PEDs: Results of the Multi-Consortium International Pediatric Brain Tumor Segmentation Challenge 2023](https://mlsanity.com/paper/2407.08855/) (2024)
- [HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts](https://mlsanity.com/paper/2601.00583/) (2026)
- [BenchECG and xECG: a benchmark and baseline for ECG foundation models](https://mlsanity.com/paper/2509.10151/) (2025)
- [A Systematic Review of Generalization Research in Medical Image Classification](https://mlsanity.com/paper/2403.12167/) (2024)
- [fastWDM3D: Fast and Accurate 3D Healthy Tissue Inpainting](https://mlsanity.com/paper/2507.13146/) (2025)
- [Model Ensemble for Brain Tumor Segmentation in Magnetic Resonance Imaging](https://mlsanity.com/paper/2409.08232/) (2024)
- [BraTS-Path Challenge: Assessing Heterogeneous Histopathologic Brain Tumor Sub-regions](https://mlsanity.com/paper/2405.10871/) (2024)
- [Diffuse-UDA: Addressing Unsupervised Domain Adaptation in Medical Image Segmentation with Appearance and Structure Aligned Diffusion Models](https://mlsanity.com/paper/2408.05985/) (2024)
- [Adult Glioma Segmentation in Sub-Saharan Africa using Transfer Learning on Stratified Finetuning Data](https://mlsanity.com/paper/2412.04111/) (2024)
- [Real-World Federated Learning in Radiology: Hurdles to overcome and Benefits to gain](https://mlsanity.com/paper/2405.09409/) (2024)
- [Analysis of the 2024 BraTS Meningioma Radiotherapy Planning Automated Segmentation Challenge](https://mlsanity.com/paper/2405.18383/) (2024)
- [MIST: A Simple and Scalable End-To-End 3D Medical Imaging Segmentation Framework](https://mlsanity.com/paper/2407.21343/) (2024)
- [CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs](https://mlsanity.com/paper/2505.10496/) (2025)
- [Rethinking Client Drift in Federated Learning: A Logit Perspective](https://mlsanity.com/paper/2308.10162/) (2023)
- [Building Universal Foundation Models for Medical Image Analysis with Spatially Adaptive Networks](https://mlsanity.com/paper/2312.07630/) (2023)
- [Advancing Brain Tumor Inpainting with Generative Models](https://mlsanity.com/paper/2402.01509/) (2024)
- [Recent Advances, Applications, and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposium](https://mlsanity.com/paper/2403.01628/) (2024)
- [On Enhancing Brain Tumor Segmentation Across Diverse Populations with Convolutional Neural Networks](https://mlsanity.com/paper/2405.02852/) (2024)
- [Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments](https://mlsanity.com/paper/2406.16791/) (2024)
- [Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives](https://mlsanity.com/paper/2502.16841/) (2025)

## Abstract (arXiv)

Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap. We call for researchers and organizations to join us in creating the MedPerf open benchmarking platform.