# Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentation

arXiv: https://arxiv.org/abs/2203.10144 · HTML: https://mlsanity.com/paper/2203.10144/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Fair Resource Allocation in Federated Learning](https://mlsanity.com/paper/1905.10497/) (2019)
- [iDLG: Improved Deep Leakage from Gradients](https://mlsanity.com/paper/2001.02610/) (2020)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [See through Gradients: Image Batch Recovery via GradInversion](https://mlsanity.com/paper/2104.07586/) (2021)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space](https://mlsanity.com/paper/2103.06030/) (2021)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)

## Built on by (most cited first, 30 of 42)

- [NVIDIA FLARE: Federated Learning from Simulation to Real-World](https://mlsanity.com/paper/2210.13291/) (2022)
- [Federated Learning for Medical Image Analysis: A Survey](https://mlsanity.com/paper/2306.05980/) (2023)
- [Learning Across Domains and Devices: Style-Driven Source-Free Domain Adaptation in Clustered Federated Learning](https://mlsanity.com/paper/2210.02326/) (2022)
- [Faster Adaptive Federated Learning](https://mlsanity.com/paper/2212.00974/) (2022)
- [FedTracker: Furnishing Ownership Verification and Traceability for Federated Learning Model](https://mlsanity.com/paper/2211.07160/) (2022)
- [Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversity](https://mlsanity.com/paper/2405.16585/) (2024)
- [Adversarial Weight Perturbation Improves Generalization in Graph Neural Networks](https://mlsanity.com/paper/2212.04983/) (2022)
- [AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images](https://mlsanity.com/paper/2303.00865/) (2023)
- [Federated Multi-organ Segmentation with Inconsistent Labels](https://mlsanity.com/paper/2206.07156/) (2022)
- [MH-pFLID: Model Heterogeneous personalized Federated Learning via Injection and Distillation for Medical Data Analysis](https://mlsanity.com/paper/2405.06822/) (2024)
- [FedUV: Uniformity and Variance for Heterogeneous Federated Learning](https://mlsanity.com/paper/2402.18372/) (2024)
- [dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysis](https://mlsanity.com/paper/2503.10412/) (2025)
- [FedDAR: Federated Domain-Aware Representation Learning](https://mlsanity.com/paper/2209.04007/) (2022)
- [A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applications](https://mlsanity.com/paper/2403.01387/) (2024)
- [Reward Systems for Trustworthy Medical Federated Learning](https://mlsanity.com/paper/2205.00470/) (2022)
- [FedSynthCT-Brain: A Federated Learning Framework for Multi-Institutional Brain MRI-to-CT Synthesis](https://mlsanity.com/paper/2412.06690/) (2024)
- [FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation](https://mlsanity.com/paper/2306.11046/) (2023)
- [Empowering Federated Learning for Massive Models with NVIDIA FLARE](https://mlsanity.com/paper/2402.07792/) (2024)
- [Federated brain tumor segmentation: an extensive benchmark](https://mlsanity.com/paper/2410.17265/) (2024)
- [Navigating Distribution Shifts in Medical Image Analysis: A Survey](https://mlsanity.com/paper/2411.05824/) (2024)
- [Scale Federated Learning for Label Set Mismatch in Medical Image Classification](https://mlsanity.com/paper/2304.06931/) (2023)
- [Bridging Data Islands: Geographic Heterogeneity-Aware Federated Learning for Collaborative Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2404.09292/) (2024)
- [Federated Client-tailored Adapter for Medical Image Segmentation](https://mlsanity.com/paper/2504.18020/) (2025)
- [Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practice](https://mlsanity.com/paper/2510.12595/) (2025)
- [Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems](https://mlsanity.com/paper/2209.10058/) (2022)
- [Benchmarking Collaborative Learning Methods Cost-Effectiveness for Prostate Segmentation](https://mlsanity.com/paper/2309.17097/) (2023)
- [FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data](https://mlsanity.com/paper/2411.07050/) (2024)
- [FedDAG: Federated Domain Adversarial Generation Towards Generalizable Medical Image Analysis](https://mlsanity.com/paper/2501.13967/) (2025)
- [Conformalized Neural Networks for Federated Uncertainty Quantification under Dual Heterogeneity](https://mlsanity.com/paper/2602.23296/) (2026)
- [ProtegoFed: Backdoor-Free Federated Instruction Tuning with Interspersed Poisoned Data](https://mlsanity.com/paper/2603.00516/) (2026)

## Abstract (arXiv)

Cross-silo federated learning (FL) has attracted much attention in medical imaging analysis with deep learning in recent years as it can resolve the critical issues of insufficient data, data privacy, and training efficiency. However, there can be a generalization gap between the model trained from FL and the one from centralized training. This important issue comes from the non-iid data distribution of the local data in the participating clients and is well-known as client drift. In this work, we propose a novel training framework FedSM to avoid the client drift issue and successfully close the generalization gap compared with the centralized training for medical image segmentation tasks for the first time. We also propose a novel personalized FL objective formulation and a new method SoftPull to solve it in our proposed framework FedSM. We conduct rigorous theoretical analysis to guarantee its convergence for optimizing the non-convex smooth objective function. Real-world medical image segmentation experiments using deep FL validate the motivations and effectiveness of our proposed method.