# Harnessing Increased Client Participation with Cohort-Parallel Federated Learning

arXiv: https://arxiv.org/abs/2405.15644 · HTML: https://mlsanity.com/paper/2405.15644/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 25 of 25)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Federated learning with hierarchical clustering of local updates to improve training on non-IID data](https://mlsanity.com/paper/2004.11791/) (2020)
- [FedScale: Benchmarking Model and System Performance of Federated Learning at Scale](https://mlsanity.com/paper/2105.11367/) (2021)
- [Papaya: Practical, Private, and Scalable Federated Learning](https://mlsanity.com/paper/2111.04877/) (2021)
- [AI Benchmark: All About Deep Learning on Smartphones in 2019](https://mlsanity.com/paper/1910.06663/) (2019)
- [On Large-Cohort Training for Federated Learning](https://mlsanity.com/paper/2106.07820/) (2021)
- [Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing](https://mlsanity.com/paper/2106.04502/) (2021)
- [Flexible Clustered Federated Learning for Client-Level Data Distribution Shift](https://mlsanity.com/paper/2108.09749/) (2021)
- [Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation](https://mlsanity.com/paper/2209.04599/) (2022)
- [Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning](https://mlsanity.com/paper/2307.09619/) (2023)
- [Resource-Efficient Federated Learning](https://mlsanity.com/paper/2111.01108/) (2021)
- [Decentralized Learning Made Easy with DecentralizePy](https://mlsanity.com/paper/2304.08322/) (2023)
- [Auxo: Efficient Federated Learning via Scalable Client Clustering](https://mlsanity.com/paper/2210.16656/) (2022)
- [TEE-based decentralized recommender systems: The raw data sharing redemption](https://mlsanity.com/paper/2202.11655/) (2022)
- [Decentralized Learning with Multi-Headed Distillation](https://mlsanity.com/paper/2211.15774/) (2022)

## Abstract (arXiv)

Federated learning (FL) is a machine learning approach where nodes collaboratively train a global model. As more nodes participate in a round of FL, the effectiveness of individual model updates by nodes also diminishes. In this study, we increase the effectiveness of client updates by dividing the network into smaller partitions, or cohorts. We introduce Cohort-Parallel Federated Learning (CPFL): a novel learning approach where each cohort independently trains a global model using FL, until convergence, and the produced models by each cohort are then unified using knowledge distillation. The insight behind CPFL is that smaller, isolated networks converge quicker than in a one-network setting where all nodes participate. Through exhaustive experiments involving realistic traces and non-IID data distributions on the CIFAR-10 and FEMNIST image classification tasks, we investigate the balance between the number of cohorts, model accuracy, training time, and compute resources. Compared to traditional FL, CPFL with four cohorts, non-IID data distribution, and CIFAR-10 yields a 1.9x reduction in train time and a 1.3x reduction in resource usage, with a minimal drop in test accuracy.