# Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients

arXiv: https://arxiv.org/abs/2201.10899 · HTML: https://mlsanity.com/paper/2201.10899/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)
- [Federated learning with hierarchical clustering of local updates to improve training on non-IID data](https://mlsanity.com/paper/2004.11791/) (2020)
- [Task2Vec: Task Embedding for Meta-Learning](https://mlsanity.com/paper/1902.03545/) (2019)
- [Federated Visual Classification with Real-World Data Distribution](https://mlsanity.com/paper/2003.08082/) (2020)
- [BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning](https://mlsanity.com/paper/1905.06731/) (2019)
- [First Analysis of Local GD on Heterogeneous Data](https://mlsanity.com/paper/1909.04715/) (2019)
- [Decentralized Federated Learning: A Segmented Gossip Approach](https://mlsanity.com/paper/1908.07782/) (2019)
- [Cluster-driven Graph Federated Learning over Multiple Domains](https://mlsanity.com/paper/2104.14628/) (2021)
- [A distillation-based approach integrating continual learning and federated learning for pervasive services](https://mlsanity.com/paper/2109.04197/) (2021)
- [FedFMC: Sequential Efficient Federated Learning on Non-iid Data](https://mlsanity.com/paper/2006.10937/) (2020)

## Built on by (most cited first, 6 of 6)

- [MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare](https://mlsanity.com/paper/2206.08516/) (2022)
- [One-Shot Sequential Federated Learning for Non-IID Data by Enhancing Local Model Diversity](https://mlsanity.com/paper/2404.12130/) (2024)
- [A Privacy Preserving System for Movie Recommendations Using Federated Learning](https://mlsanity.com/paper/2303.04689/) (2023)
- [Federated Source-free Domain Adaptation for Classification: Weighted Cluster Aggregation for Unlabeled Data](https://mlsanity.com/paper/2412.13757/) (2024)
- [Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentum](https://mlsanity.com/paper/2311.18578/) (2023)
- [Sequential Federated Learning in Hierarchical Architecture on Non-IID Datasets](https://mlsanity.com/paper/2408.09762/) (2024)

## Abstract (arXiv)

Federated Learning (FL) allows training machine learning models in privacy-constrained scenarios by enabling the cooperation of edge devices without requiring local data sharing. This approach raises several challenges due to the different statistical distribution of the local datasets and the clients' computational heterogeneity. In particular, the presence of highly non-i.i.d. data severely impairs both the performance of the trained neural network and its convergence rate, increasing the number of communication rounds requested to reach a performance comparable to that of the centralized scenario. As a solution, we propose FedSeq, a novel framework leveraging the sequential training of subgroups of heterogeneous clients, i.e. superclients, to emulate the centralized paradigm in a privacy-compliant way. Given a fixed budget of communication rounds, we show that FedSeq outperforms or match several state-of-the-art federated algorithms in terms of final performance and speed of convergence. Finally, our method can be easily integrated with other approaches available in the literature. Empirical results show that combining existing algorithms with FedSeq further improves its final performance and convergence speed. We test our method on CIFAR-10 and CIFAR-100 and prove its effectiveness in both i.i.d. and non-i.i.d. scenarios.