# FedMedICL: Towards Holistic Evaluation of Distribution Shifts in Federated Medical Imaging

arXiv: https://arxiv.org/abs/2407.08822 · HTML: https://mlsanity.com/paper/2407.08822/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 35 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 15 of 15)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings](https://mlsanity.com/paper/2210.04620/) (2022)
- [Change is Hard: A Closer Look at Subpopulation Shift](https://mlsanity.com/paper/2302.12254/) (2023)
- [Don't forget, there is more than forgetting: new metrics for Continual Learning](https://mlsanity.com/paper/1810.13166/) (2018)
- [Long-Tailed Recognition via Weight Balancing](https://mlsanity.com/paper/2203.14197/) (2022)
- [COVID-19 Image Data Collection: Prospective Predictions Are the Future](https://mlsanity.com/paper/2006.11988/) (2020)
- [MEDFAIR: Benchmarking Fairness for Medical Imaging](https://mlsanity.com/paper/2210.01725/) (2022)
- [Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification](https://mlsanity.com/paper/1909.01940/) (2019)
- [LifeLonger: A Benchmark for Continual Disease Classification](https://mlsanity.com/paper/2204.05737/) (2022)

## Built on by (most cited first, 5 of 5)

- [Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey](https://mlsanity.com/paper/2412.13840/) (2024)
- [Federated Learning in Chemical Engineering: A Tutorial on a Framework for Privacy-Preserving Collaboration Across Distributed Data Sources](https://mlsanity.com/paper/2411.16737/) (2024)
- [Federated Learning for Large Models in Medical Imaging: A Comprehensive Review](https://mlsanity.com/paper/2508.20414/) (2025)
- [Prototype-Guided and Lightweight Adapters for Inherent Interpretation and Generalisation in Federated Learning](https://mlsanity.com/paper/2507.05852/) (2025)
- [Foundation Models for Medical Imaging: Status, Challenges, and Directions](https://mlsanity.com/paper/2602.15913/) (2026)

## Abstract (arXiv)

For medical imaging AI models to be clinically impactful, they must generalize. However, this goal is hindered by (i) diverse types of distribution shifts, such as temporal, demographic, and label shifts, and (ii) limited diversity in datasets that are siloed within single medical institutions. While these limitations have spurred interest in federated learning, current evaluation benchmarks fail to evaluate different shifts simultaneously. However, in real healthcare settings, multiple types of shifts co-exist, yet their impact on medical imaging performance remains unstudied. In response, we introduce FedMedICL, a unified framework and benchmark to holistically evaluate federated medical imaging challenges, simultaneously capturing label, demographic, and temporal distribution shifts. We comprehensively evaluate several popular methods on six diverse medical imaging datasets (totaling 550 GPU hours). Furthermore, we use FedMedICL to simulate COVID-19 propagation across hospitals and evaluate whether methods can adapt to pandemic changes in disease prevalence. We find that a simple batch balancing technique surpasses advanced methods in average performance across FedMedICL experiments. This finding questions the applicability of results from previous, narrow benchmarks in real-world medical settings.