mlsanityData to 7 Oct 2026

Harnessing Increased Client Participation with Cohort-Parallel Federated Learning

arXiv 2405.15644 ↗First posted 24 May 2024cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a machine learning approach where nodes collaboratively train a global model. As more nodes participate in a round of FL, the effectiveness of individual model updates by nodes also diminishes. In this study, we increase the effectiveness of client updates by dividing the network into smaller partitions, or cohorts. We introduce Cohort-Parallel Federated Learning (CPFL): a novel learning approach where each cohort independently trains a global model using FL, until convergence, and the produced models by each cohort are then unified using knowledge distillation. The insight behind CPFL is that smaller, isolated networks converge quicker than in a one-network setting where all nodes participate. Through exhaustive experiments involving realistic traces and non-IID data distributions on the CIFAR-10 and FEMNIST image classification tasks, we investigate the balance between the number of cohorts, model accuracy, training time, and compute resources. Compared to traditional FL, CPFL with four cohorts, non-IID data distribution, and CIFAR-10 yields a 1.9x reduction in train time and a 1.3x reduction in resource usage, with a minimal drop in test accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 40 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

25 of 40 references.

References · 25

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  5. 2019Towards Federated Learning at Scale: System Designcited by 826
  6. 2018LEAF: A Benchmark for Federated Settingscited by 780
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  8. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  9. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  10. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  11. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  12. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 222
13 more
  1. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  2. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 89
  3. 2019AI Benchmark: All About Deep Learning on Smartphones in 2019cited by 76
  4. 2021On Large-Cohort Training for Federated Learningcited by 73
  5. 2021Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharingcited by 58
  6. 2021Flexible Clustered Federated Learning for Client-Level Data Distribution Shiftcited by 47
  7. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
  8. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  9. 2021Resource-Efficient Federated Learningcited by 26
  10. 2023Decentralized Learning Made Easy with DecentralizePycited by 18
  11. 2022Auxo: Efficient Federated Learning via Scalable Client Clusteringcited by 8
  12. 2022TEE-based decentralized recommender systems: The raw data sharing redemptioncited by 6
  13. 2022Decentralized Learning with Multi-Headed Distillationcited by 5

Built on by

Not cited yet.