mlsanityData to 7 Oct 2026

MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation

arXiv 2110.01406 ↗First posted 29 Sep 2021cs.LG cs.DC cs.PFcited by 88

Not on the map. Why?

Abstract (arXiv)

Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap. We call for researchers and organizations to join us in creating the MedPerf open benchmarking platform.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 24 of its 143 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

24 of 143 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  4. 2020The Future of Digital Health with Federated Learningcited by 625
  5. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  6. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  7. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  8. 2018fastMRI: An Open Dataset and Benchmarks for Accelerated MRIcited by 464
  9. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
  10. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 326
  11. 2020CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentationcited by 307
  12. 2016Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentationcited by 263
12 more
  1. 2019MLPerf Inference Benchmarkcited by 195
  2. 2019Causality matters in medical imagingcited by 193
  3. 2018Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentationcited by 148
  4. 2020TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networkscited by 139
  5. 2019MLPerf Training Benchmarkcited by 114
  6. 2018Why rankings of biomedical image analysis competitions should be interpreted with carecited by 90
  7. 2021Common Limitations of Image Processing Metrics: A Picture Storycited by 58
  8. 2021OpenFL: An open-source framework for Federated Learningcited by 58
  9. 2021The Federated Tumor Segmentation (FeTS) Challengecited by 49
  10. 2021GaNDLF: A Generally Nuanced Deep Learning Framework for Scalable End-to-End Clinical Workflows in Medical Imagingcited by 25
  11. 2021Spark NLP: Natural Language Understanding at Scalecited by 16
  12. 2020Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 30; 2025: 28; 2026: 26.

88 papers.

Sort:

By area:

  1. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 79
  2. 2024WDM: 3D Wavelet Diffusion Models for High-Resolution Medical Image Synthesiscited by 56
  3. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  4. 2025nnInteractive: Redefining 3D Promptable Segmentationcited by 39
  5. 2024Denoising Diffusion Models for 3D Healthy Brain Tissue Inpaintingcited by 22
  6. 2024Analysis of the BraTS 2023 Intracranial Meningioma Segmentation Challengecited by 14
  7. 2024Denoising Diffusion Models for Inpainting of Healthy Brain Tissuecited by 13
  8. 2024Advanced Tumor Segmentation in Medical Imaging: An Ensemble Approach for BraTS 2023 Adult Glioma and Pediatric Tumor Taskscited by 11
  9. 2024BraTS-PEDs: Results of the Multi-Consortium International Pediatric Brain Tumor Segmentation Challenge 2023cited by 11
  10. 2025LesionLocator: Zero-Shot Universal Tumor Segmentation and Tracking in 3D Whole-Body Imagingcited by 11
  11. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Textcited by 11
  12. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
38 more
  1. 2024A Systematic Review of Generalization Research in Medical Image Classificationcited by 8
  2. 2025fastWDM3D: Fast and Accurate 3D Healthy Tissue Inpaintingcited by 8
  3. 2025BenchECG and xECG: a benchmark and baseline for ECG foundation modelscited by 8
  4. 2024Model Ensemble for Brain Tumor Segmentation in Magnetic Resonance Imagingcited by 7
  5. 2024BraTS-Path Challenge: Assessing Heterogeneous Histopathologic Brain Tumor Sub-regionscited by 6
  6. 2024Diffuse-UDA: Addressing Unsupervised Domain Adaptation in Medical Image Segmentation with Appearance and Structure Aligned Diffusion Modelscited by 6
  7. 2024Adult Glioma Segmentation in Sub-Saharan Africa using Transfer Learning on Stratified Finetuning Datacited by 6
  8. 2024Real-World Federated Learning in Radiology: Hurdles to overcome and Benefits to gaincited by 5
  9. 2024Analysis of the 2024 BraTS Meningioma Radiotherapy Planning Automated Segmentation Challengecited by 5
  10. 2024MIST: A Simple and Scalable End-To-End 3D Medical Imaging Segmentation Frameworkcited by 5
  11. 2025CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographscited by 5
  12. 2023Building Universal Foundation Models for Medical Image Analysis with Spatially Adaptive Networkscited by 4
  13. 2024Advancing Brain Tumor Inpainting with Generative Modelscited by 4
  14. 2023Rethinking Client Drift in Federated Learning: A Logit Perspectivecited by 3
  15. 2024Recent Advances, Applications, and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposiumcited by 3
  16. 2024On Enhancing Brain Tumor Segmentation Across Diverse Populations with Convolutional Neural Networkscited by 3
  17. 2024Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournamentscited by 3
  18. 2025Fair Foundation Models for Medical Image Analysis: Challenges and Perspectivescited by 3
  19. 2025AI Benchmark Democratization and Carpentrycited by 3
  20. 2024Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Studycited by 2
  21. 2024Transferring Knowledge from High-Quality to Low-Quality MRI for Adult Glioma Diagnosiscited by 2
  22. 2024Non-Convex Optimization in Federated Learning via Variance Reduction and Adaptive Learningcited by 2
  23. 2025The Missing Piece: A Case for Pre-Training in 3D Medical Object Detectioncited by 2
  24. 2025Auto3DSeg for Brain Tumor Segmentation from 3D MRI in BraTS 2023 Challengecited by 2
  25. 2025An MLCommons Scientific Benchmarks Ontologycited by 2
  26. 2025The MICCAI Federated Tumor Segmentation (FeTS) Challenge 2024: Efficient and Robust Aggregation Methods for Federated Learningcited by 2
  27. 2023A Comprehensive View of Personalized Federated Learning on Heterogeneous Clinical Datasetscited by 1
  28. 2024Statistical Test for Diffusion-Based Anomaly Localization via Selective Inferencecited by 1
  29. 2024Attention-Enhanced Hybrid Feature Aggregation Network for 3D Brain Tumor Segmentationcited by 1
  30. 2024Segmentation of Pediatric Brain Tumors using a Radiologically informed, Deep Learning Cascadecited by 1
  31. 2024Parameter-efficient Fine-tuning for improved Convolutional Baseline for Brain Tumor Segmentation in Sub-Saharan Africa Adult Glioma Datasetcited by 1
  32. 2025Federated learning, ethics, and the double black box problem in medical AIcited by 1
  33. 2025Multi-modal brain MRI synthesis based on SwinUNETRcited by 1
  34. 2025Synthesizing Accurate and Realistic T1-weighted Contrast-Enhanced MR Images using Posterior-Mean Rectified Flowcited by 1
  35. 2025DSFL: A Dual-Server Byzantine-Resilient Federated Learning Framework via Group-Based Secure Aggregationcited by 1
  36. 2026Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcarecited by 1
  37. 2026GEN-Guard: Correcting Generalization Failures for Deployable Federated Surgical AIcited by 1
  38. 2023All Sizes Matter: Improving Volumetric Brain Segmentation on Small Lesionscited by 0

50 most cited of 88 All 88 (JSON)