# Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization

arXiv: https://arxiv.org/abs/2309.06692 · HTML: https://mlsanity.com/paper/2309.06692/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 30 of 30)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [Personalized Cross-Silo Federated Learning on Non-IID Data](https://mlsanity.com/paper/2007.03797/) (2020)
- [Generalized Federated Learning via Sharpness Aware Minimization](https://mlsanity.com/paper/2206.02618/) (2022)
- [SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum](https://mlsanity.com/paper/1910.00643/) (2019)
- [Improving Generalization in Federated Learning by Seeking Flat Minima](https://mlsanity.com/paper/2203.11834/) (2022)
- [Layer-wised Model Aggregation for Personalized Federated Learning](https://mlsanity.com/paper/2205.03993/) (2022)
- [Divergence-aware Federated Self-Supervised Learning](https://mlsanity.com/paper/2204.04385/) (2022)
- [Collaborative Unsupervised Visual Representation Learning from Decentralized Data](https://mlsanity.com/paper/2108.06492/) (2021)
- [HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images](https://mlsanity.com/paper/2112.10775/) (2021)
- [Domain Generalization via Gradient Surgery](https://mlsanity.com/paper/2108.01621/) (2021)
- [Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning](https://mlsanity.com/paper/2210.00226/) (2022)
- [FedX: Unsupervised Federated Learning with Cross Knowledge Distillation](https://mlsanity.com/paper/2207.09158/) (2022)
- [FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reduction](https://mlsanity.com/paper/2205.13462/) (2022)
- [Federated Learning for Face Recognition with Gradient Correction](https://mlsanity.com/paper/2112.07246/) (2021)

## Built on by (most cited first, 4 of 4)

- [Tackling Data Heterogeneity in Federated Learning via Loss Decomposition](https://mlsanity.com/paper/2408.12300/) (2024)
- [Federated and Transfer Learning for Cancer Detection Based on Image Analysis](https://mlsanity.com/paper/2405.20126/) (2024)
- [HLF-FSL. A Decentralized Federated Split Learning Solution for IoT on Hyperledger Fabric](https://mlsanity.com/paper/2507.07637/) (2025)
- [FedCKA: Representation-Guided Layer Personalization for Federated 3D Perception Across Driving Domains](https://mlsanity.com/paper/2610.01510/) (2026)

## Abstract (arXiv)

Federated learning (FL) is a privacy-preserving paradigm for collaboratively training a global model from decentralized clients. However, the performance of FL is hindered by non-independent and identically distributed (non-IID) data and device heterogeneity. In this work, we revisit this key challenge through the lens of gradient conflicts on the server side. Specifically, we first investigate the gradient conflict phenomenon among multiple clients and reveal that stronger heterogeneity leads to more severe gradient conflicts. To tackle this issue, we propose FedGH, a simple yet effective method that mitigates local drifts through Gradient Harmonization. This technique projects one gradient vector onto the orthogonal plane of the other within conflicting client pairs. Extensive experiments demonstrate that FedGH consistently enhances multiple state-of-the-art FL baselines across diverse benchmarks and non-IID scenarios. Notably, FedGH yields more significant improvements in scenarios with stronger heterogeneity. As a plug-and-play module, FedGH can be seamlessly integrated into any FL framework without requiring hyperparameter tuning.