Learning from straggler clients in federated learning
Not on the map. Why?
Abstract (arXiv)
How well do existing federated learning algorithms learn from client devices that return model updates with a significant time delay? Is it even possible to learn effectively from clients that report back minutes, hours, or days after being scheduled? We answer these questions by developing Monte Carlo simulations of client latency that are guided by real-world applications. We study synchronous optimization algorithms like FedAvg and FedAdam as well as the asynchronous FedBuff algorithm, and observe that all these existing approaches struggle to learn from severely delayed clients. To improve upon this situation, we experiment with modifications, including distillation regularization and exponential moving averages of model weights. Finally, we introduce two new algorithms, FARe-DUST and FeAST-on-MSG, based on distillation and averaging, respectively. Experiments with the EMNIST, CIFAR-100, and StackOverflow benchmark federated learning tasks demonstrate that our new algorithms outperform existing ones in terms of accuracy for straggler clients, while also providing better trade-offs between training time and total accuracy.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 56 references that are in the map. First posted on arXiv 14 Mar 2024.
Builds on
29 of 56 references.
References · 29
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2020Adaptive Federated Optimizationcited by 1,036
- 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
- 2019Towards Federated Learning at Scale: System Designcited by 826
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
- 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
- 2021A Field Guide to Federated Optimizationcited by 312
- 2019Asynchronous Federated Optimizationcited by 264
- 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
17 more
- 2018Large scale distributed neural network training through online distillationcited by 144
- 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
- 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 98
- 2022Sharpness-Aware Training for Freecited by 95
- 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 89
- 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
- 2021On Large-Cohort Training for Federated Learningcited by 73
- 2015Perturbed Iterate Analysis for Asynchronous Stochastic Optimizationcited by 67
- 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
- 2021FedJAX: Federated learning simulation with JAXcited by 32
- 2022AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregationcited by 19
- 2019The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communicationcited by 12
- 2021Secure Aggregation for Buffered Asynchronous Federated Learningcited by 10
- 2022Production federated keyword spotting via distillation, filtering, and joint federated-centralized trainingcited by 9
- 2020A Closer Look at Codistillation for Distributed Trainingcited by 8
- 2022Mixed Federated Learning: Joint Decentralized and Centralized Learningcited by 8
- 2022Recycling Scraps: Improving Private Learning by Leveraging Intermediate Checkpointscited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.