# HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images

arXiv: https://arxiv.org/abs/2112.10775 · HTML: https://mlsanity.com/paper/2112.10775/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 20 Dec 2021.

## Builds on (most cited first, 28 of 28)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [SCAFFOLD: Stochastic Controlled Averaging for Federated Learning](https://mlsanity.com/paper/1910.06378/) (2019)
- [Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation](https://mlsanity.com/paper/1810.04304/) (2018)
- [Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron](https://mlsanity.com/paper/1810.07288/) (2018)
- [MS-Net: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Data](https://mlsanity.com/paper/2002.03366/) (2020)
- [Robust Federated Learning: The Case of Affine Distribution Shifts](https://mlsanity.com/paper/2006.08907/) (2020)
- [Federated Learning for Breast Density Classification: A Real-World Implementation](https://mlsanity.com/paper/2009.01871/) (2020)
- [Siloed Federated Learning for Multi-Centric Histopathology Datasets](https://mlsanity.com/paper/2008.07424/) (2020)
- [Inverse Distance Aggregation for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2008.07665/) (2020)
- [Federated Transfer Learning for EEG Signal Classification](https://mlsanity.com/paper/2004.12321/) (2020)
- [HHHFL: Hierarchical Heterogeneous Horizontal Federated Learning for Electroencephalography](https://mlsanity.com/paper/1909.05784/) (2019)
- [Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data](https://mlsanity.com/paper/2009.06557/) (2020)
- [Fed-Sim: Federated Simulation for Medical Imaging](https://mlsanity.com/paper/2009.00668/) (2020)

## Built on by (most cited first, 30 of 64)

- [Federated Learning for Medical Image Analysis: A Survey](https://mlsanity.com/paper/2306.05980/) (2023)
- [Federated Domain Generalization for Image Recognition via Cross-Client Style Transfer](https://mlsanity.com/paper/2210.00912/) (2022)
- [Eliminating Domain Bias for Federated Learning in Representation Space](https://mlsanity.com/paper/2311.14975/) (2023)
- [Think Twice Before Selection: Federated Evidential Active Learning for Medical Image Analysis with Domain Shifts](https://mlsanity.com/paper/2312.02567/) (2023)
- [GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting](https://mlsanity.com/paper/2302.14307/) (2023)
- [FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation](https://mlsanity.com/paper/2205.14900/) (2022)
- [IOP-FL: Inside-Outside Personalization for Federated Medical Image Segmentation](https://mlsanity.com/paper/2204.08467/) (2022)
- [FedFA: Federated Feature Augmentation](https://mlsanity.com/paper/2301.12995/) (2023)
- [Federated Learning for Computer Vision](https://mlsanity.com/paper/2308.13558/) (2023)
- [FedIIC: Towards Robust Federated Learning for Class-Imbalanced Medical Image Classification](https://mlsanity.com/paper/2206.13803/) (2022)
- [A Simple Data Augmentation for Feature Distribution Skewed Federated Learning](https://mlsanity.com/paper/2306.09363/) (2023)
- [From Challenges and Pitfalls to Recommendations and Opportunities: Implementing Federated Learning in Healthcare](https://mlsanity.com/paper/2409.09727/) (2024)
- [Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions](https://mlsanity.com/paper/2411.12377/) (2024)
- [FedCiR: Client-Invariant Representation Learning for Federated Non-IID Features](https://mlsanity.com/paper/2308.15786/) (2023)
- [Resilient Contrastive Pre-training under Non-Stationary Drift](https://mlsanity.com/paper/2502.07620/) (2025)
- [Federated Learning with Domain Shift Eraser](https://mlsanity.com/paper/2503.13063/) (2025)
- [CADIS: Handling Cluster-skewed Non-IID Data in Federated Learning with Clustered Aggregation and Knowledge DIStilled Regularization](https://mlsanity.com/paper/2302.10413/) (2023)
- [Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstruction](https://mlsanity.com/paper/2308.10910/) (2023)
- [Feasibility of Federated Learning from Client Databases with Different Brain Diseases and MRI Modalities](https://mlsanity.com/paper/2406.11636/) (2024)
- [Navigating Distribution Shifts in Medical Image Analysis: A Survey](https://mlsanity.com/paper/2411.05824/) (2024)
- [Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization](https://mlsanity.com/paper/2309.06692/) (2023)
- [PPIDSG: A Privacy-Preserving Image Distribution Sharing Scheme with GAN in Federated Learning](https://mlsanity.com/paper/2312.10380/) (2023)
- [Open-Vocabulary Federated Learning with Multimodal Prototyping](https://mlsanity.com/paper/2404.01232/) (2024)
- [Federated Learning under Partially Class-Disjoint Data via Manifold Reshaping](https://mlsanity.com/paper/2405.18983/) (2024)
- [Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data](https://mlsanity.com/paper/2504.14694/) (2025)
- [FedSSO: A Federated Server-Side Second-Order Optimization Algorithm](https://mlsanity.com/paper/2206.09576/) (2022)
- [Scale Federated Learning for Label Set Mismatch in Medical Image Classification](https://mlsanity.com/paper/2304.06931/) (2023)
- [FedSODA: Federated Cross-assessment and Dynamic Aggregation for Histopathology Segmentation](https://mlsanity.com/paper/2312.12824/) (2023)
- [Memory-Efficient Prompt Tuning for Incremental Histopathology Classification](https://mlsanity.com/paper/2401.11674/) (2024)
- [Federated Client-tailored Adapter for Medical Image Segmentation](https://mlsanity.com/paper/2504.18020/) (2025)

## Abstract (arXiv)

Multiple medical institutions collaboratively training a model using federated learning (FL) has become a promising solution for maximizing the potential of data-driven models, yet the non-independent and identically distributed (non-iid) data in medical images is still an outstanding challenge in real-world practice. The feature heterogeneity caused by diverse scanners or protocols introduces a drift in the learning process, in both local (client) and global (server) optimizations, which harms the convergence as well as model performance. Many previous works have attempted to address the non-iid issue by tackling the drift locally or globally, but how to jointly solve the two essentially coupled drifts is still unclear. In this work, we concentrate on handling both local and global drifts and introduce a new harmonizing framework called HarmoFL. First, we propose to mitigate the local update drift by normalizing amplitudes of images transformed into the frequency domain to mimic a unified imaging setting, in order to generate a harmonized feature space across local clients. Second, based on harmonized features, we design a client weight perturbation guiding each local model to reach a flat optimum, where a neighborhood area of the local optimal solution has a uniformly low loss. Without any extra communication cost, the perturbation assists the global model to optimize towards a converged optimal solution by aggregating several local flat optima. We have theoretically analyzed the proposed method and empirically conducted extensive experiments on three medical image classification and segmentation tasks, showing that HarmoFL outperforms a set of recent state-of-the-art methods with promising convergence behavior. Code is available at https://github.com/med-air/HarmoFL.