# Federated learning with hierarchical clustering of local updates to improve training on non-IID data

arXiv: https://arxiv.org/abs/2004.11791 · HTML: https://mlsanity.com/paper/2004.11791/

## Key facts

- Cited by 223 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 30 of 223)

- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [FedBABU: Towards Enhanced Representation for Federated Image Classification](https://mlsanity.com/paper/2106.06042/) (2021)
- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [Federated Learning for Open Banking](https://mlsanity.com/paper/2108.10749/) (2021)
- [Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges](https://mlsanity.com/paper/2211.08413/) (2022)
- [Federated Graph Classification over Non-IID Graphs](https://mlsanity.com/paper/2106.13423/) (2021)
- [Improving Generalization in Federated Learning by Seeking Flat Minima](https://mlsanity.com/paper/2203.11834/) (2022)
- [FedProc: Prototypical Contrastive Federated Learning on Non-IID data](https://mlsanity.com/paper/2109.12273/) (2021)
- [From Distributed Machine Learning to Federated Learning: A Survey](https://mlsanity.com/paper/2104.14362/) (2021)
- [Flexible Clustered Federated Learning for Client-Level Data Distribution Shift](https://mlsanity.com/paper/2108.09749/) (2021)
- [FedSoft: Soft Clustered Federated Learning with Proximal Local Updating](https://mlsanity.com/paper/2112.06053/) (2021)
- [pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning](https://mlsanity.com/paper/2206.03655/) (2022)
- [Communication Efficiency in Federated Learning: Achievements and Challenges](https://mlsanity.com/paper/2107.10996/) (2021)
- [Efficient Personalized Federated Learning via Sparse Model-Adaptation](https://mlsanity.com/paper/2305.02776/) (2023)
- [Non-IID data and Continual Learning processes in Federated Learning: A long road ahead](https://mlsanity.com/paper/2111.13394/) (2021)
- [Federated Learning under Distributed Concept Drift](https://mlsanity.com/paper/2206.00799/) (2022)
- [Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning](https://mlsanity.com/paper/2102.12920/) (2021)
- [Mobility-Aware Cluster Federated Learning in Hierarchical Wireless Networks](https://mlsanity.com/paper/2108.09103/) (2021)
- [FedPop: A Bayesian Approach for Personalised Federated Learning](https://mlsanity.com/paper/2206.03611/) (2022)
- [On the Convergence of Clustered Federated Learning](https://mlsanity.com/paper/2202.06187/) (2022)
- [Personalized Federated Learning With Graph](https://mlsanity.com/paper/2203.00829/) (2022)
- [Federated Mixture of Experts](https://mlsanity.com/paper/2107.06724/) (2021)
- [Privacy and Fairness in Federated Learning: on the Perspective of Trade-off](https://mlsanity.com/paper/2306.14123/) (2023)
- [FedAC: An Adaptive Clustered Federated Learning Framework for Heterogeneous Data](https://mlsanity.com/paper/2403.16460/) (2024)
- [Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching](https://mlsanity.com/paper/2203.12285/) (2022)
- [A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy](https://mlsanity.com/paper/2302.10637/) (2023)
- [Personalized Federated Learning with Contextualized Generalization](https://mlsanity.com/paper/2106.13044/) (2021)
- [Topology-aware Federated Learning in Edge Computing: A Comprehensive Survey](https://mlsanity.com/paper/2302.02573/) (2023)

## Abstract (arXiv)

Federated learning (FL) is a well established method for performing machine learning tasks over massively distributed data. However in settings where data is distributed in a non-iid (not independent and identically distributed) fashion -- as is typical in real world situations -- the joint model produced by FL suffers in terms of test set accuracy and/or communication costs compared to training on iid data. We show that learning a single joint model is often not optimal in the presence of certain types of non-iid data. In this work we present a modification to FL by introducing a hierarchical clustering step (FL+HC) to separate clusters of clients by the similarity of their local updates to the global joint model. Once separated, the clusters are trained independently and in parallel on specialised models. We present a robust empirical analysis of the hyperparameters for FL+HC for several iid and non-iid settings. We show how FL+HC allows model training to converge in fewer communication rounds (significantly so under some non-iid settings) compared to FL without clustering. Additionally, FL+HC allows for a greater percentage of clients to reach a target accuracy compared to standard FL. Finally we make suggestions for good default hyperparameters to promote superior performing specialised models without modifying the the underlying federated learning communication protocol.