# Flexible Clustered Federated Learning for Client-Level Data Distribution Shift

arXiv: https://arxiv.org/abs/2108.09749 · HTML: https://mlsanity.com/paper/2108.09749/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 22 Aug 2021.

## Builds on (most cited first, 27 of 27)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Federated Machine Learning: Concept and Applications](https://mlsanity.com/paper/1902.04885/) (2019)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Federated learning with hierarchical clustering of local updates to improve training on non-IID data](https://mlsanity.com/paper/2004.11791/) (2020)
- [Client-Edge-Cloud Hierarchical Federated Learning](https://mlsanity.com/paper/1905.06641/) (2019)
- [Communication Efficient Distributed Optimization using an Approximate Newton-type Method](https://mlsanity.com/paper/1312.7853/) (2013)
- [The Effect of Natural Distribution Shift on Question Answering Models](https://mlsanity.com/paper/2004.14444/) (2020)
- [Astraea: Self-balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applications](https://mlsanity.com/paper/1907.01132/) (2019)
- [TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture](https://mlsanity.com/paper/2012.03214/) (2020)
- [On perfect clustering of high dimension, low sample size data](https://mlsanity.com/paper/1612.09121/) (2016)
- [Collaborative City Digital Twin For Covid-19 Pandemic: A Federated Learning Solution](https://mlsanity.com/paper/2011.02883/) (2020)

## Built on by (most cited first, 30 of 47)

- [Personalized Federated Learning with Feature Alignment and Classifier Collaboration](https://mlsanity.com/paper/2306.11867/) (2023)
- [Advances and Challenges in Meta-Learning: A Technical Review](https://mlsanity.com/paper/2307.04722/) (2023)
- [Federated Learning under Distributed Concept Drift](https://mlsanity.com/paper/2206.00799/) (2022)
- [L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2307.07393/) (2023)
- [FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning](https://mlsanity.com/paper/2404.02478/) (2024)
- [Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching](https://mlsanity.com/paper/2203.12285/) (2022)
- [A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy](https://mlsanity.com/paper/2302.10637/) (2023)
- [Exploiting Label Skews in Federated Learning with Model Concatenation](https://mlsanity.com/paper/2312.06290/) (2023)
- [Personalized Federated Learning via Feature Distribution Adaptation](https://mlsanity.com/paper/2411.00329/) (2024)
- [Auxo: Efficient Federated Learning via Scalable Client Clustering](https://mlsanity.com/paper/2210.16656/) (2022)
- [Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift](https://mlsanity.com/paper/2410.18478/) (2024)
- [A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Design](https://mlsanity.com/paper/2307.15245/) (2023)
- [Proximity-based Self-Federated Learning](https://mlsanity.com/paper/2407.12410/) (2024)
- [StoCFL: A Stochastically Clustered Federated Learning Framework for Non-IID Data with Dynamic Client Participation](https://mlsanity.com/paper/2303.00897/) (2023)
- [Federated Behavioural Planes: Explaining the Evolution of Client Behaviour in Federated Learning](https://mlsanity.com/paper/2405.15632/) (2024)
- [Enhancing Clustered Federated Learning: Integration of Strategies and Improved Methodologies](https://mlsanity.com/paper/2310.05397/) (2023)
- [FedD2S: Personalized Data-Free Federated Knowledge Distillation](https://mlsanity.com/paper/2402.10846/) (2024)
- [FedBaF: Federated Learning Aggregation Biased by a Foundation Model](https://mlsanity.com/paper/2410.18352/) (2024)
- [Federated Learning Clients Clustering with Adaptation to Data Drifts](https://mlsanity.com/paper/2411.01580/) (2024)
- [Federated Two Stage Decoupling With Adaptive Personalization Layers](https://mlsanity.com/paper/2308.15821/) (2023)
- [Collaborative Visual Place Recognition through Federated Learning](https://mlsanity.com/paper/2404.13324/) (2024)
- [Distributionally Robust Clustered Federated Learning: A Case Study in Healthcare](https://mlsanity.com/paper/2410.07039/) (2024)
- [A survey on Clustered Federated Learning: Taxonomy, Analysis and Applications](https://mlsanity.com/paper/2501.17512/) (2025)
- [FedTrans: Efficient Federated Learning via Multi-Model Transformation](https://mlsanity.com/paper/2404.13515/) (2024)
- [On-Demand Model and Client Deployment in Federated Learning with Deep Reinforcement Learning](https://mlsanity.com/paper/2405.07175/) (2024)
- [Federated Cross-Training Learners for Robust Generalization under Data Heterogeneity](https://mlsanity.com/paper/2405.20046/) (2024)
- [Privacy-Preserving Dataset Combination](https://mlsanity.com/paper/2502.05765/) (2025)
- [Distilling A Universal Expert from Clustered Federated Learning](https://mlsanity.com/paper/2506.20285/) (2025)
- [CLoVE: Personalized Federated Learning through Clustering of Loss Vector Embeddings](https://mlsanity.com/paper/2506.22427/) (2025)
- [Failure-tolerant Distributed Learning for Anomaly Detection in Wireless Networks](https://mlsanity.com/paper/2303.13015/) (2023)

## Abstract (arXiv)

Federated Learning (FL) enables the multiple participating devices to collaboratively contribute to a global neural network model while keeping the training data locally. Unlike the centralized training setting, the non-IID, imbalanced (statistical heterogeneity) and distribution shifted training data of FL is distributed in the federated network, which will increase the divergences between the local models and the global model, further degrading performance. In this paper, we propose a flexible clustered federated learning (CFL) framework named FlexCFL, in which we 1) group the training of clients based on the similarities between the clients' optimization directions for lower training divergence; 2) implement an efficient newcomer device cold start mechanism for framework scalability and practicality; 3) flexibly migrate clients to meet the challenge of client-level data distribution shift. FlexCFL can achieve improvements by dividing joint optimization into groups of sub-optimization and can strike a balance between accuracy and communication efficiency in the distribution shift environment. The convergence and complexity are analyzed to demonstrate the efficiency of FlexCFL. We also evaluate FlexCFL on several open datasets and made comparisons with related CFL frameworks. The results show that FlexCFL can significantly improve absolute test accuracy by +10.6% on FEMNIST compared to FedAvg, +3.5% on FashionMNIST compared to FedProx, +8.4% on MNIST compared to FeSEM. The experiment results show that FlexCFL is also communication efficient in the distribution shift environment.