mlsanityData to 14 Oct 2026

Fast Federated Learning in the Presence of Arbitrary Device Unavailability

arXiv 2106.04159 ↗First posted 8 Jun 2021cs.LG math.OCcited by 72

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) coordinates with numerous heterogeneous devices to collaboratively train a shared model while preserving user privacy. Despite its multiple advantages, FL faces new challenges. One challenge arises when devices drop out of the training process beyond the control of the central server. In this case, the convergence of popular FL algorithms such as FedAvg is severely influenced by the straggling devices. To tackle this challenge, we study federated learning algorithms under arbitrary device unavailability and propose an algorithm named Memory-augmented Impatient Federated Averaging (MIFA). Our algorithm efficiently avoids excessive latency induced by inactive devices, and corrects the gradient bias using the memorized latest updates from the devices. We prove that MIFA achieves minimax optimal convergence rates on non-i.i.d. data for both strongly convex and non-convex smooth functions. We also provide an explicit characterization of the improvement over baseline algorithms through a case study, and validate the results by numerical experiments on real-world datasets.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

33 of 59 references.

References · 33

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  5. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  6. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  7. 2019Towards Federated Learning at Scale: System Designcited by 832
  8. 2017Federated Multi-Task Learningcited by 672
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  10. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  11. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  12. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
21 more
  1. 2017How to Escape Saddle Points Efficientlycited by 285
  2. 2019Asynchronous Federated Optimizationcited by 267
  3. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  4. 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 195
  5. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  6. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  7. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  8. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 116
  9. 2018Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithmscited by 106
  10. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 78
  11. 2020Towards Flexible Device Participation in Federated Learningcited by 65
  12. 2020Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneitycited by 56
  13. 2018A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updatescited by 47
  14. 2020Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guaranteescited by 46
  15. 2019Semi-Cyclic Stochastic Gradient Descentcited by 40
  16. 2015An Asynchronous Mini-Batch Algorithm for Regularized Stochastic Optimizationcited by 38
  17. 2016Accelerated Methods for Non-Convex Optimizationcited by 20
  18. 2020Distributed Non-Convex Optimization with Sublinear Speedup under Intermittent Client Availabilitycited by 20
  19. 2019The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communicationcited by 14
  20. 2018Taming Convergence for Asynchronous Stochastic Gradient Descent with Unbounded Delay in Non-Convex Learningcited by 9
  21. 2016Analysis and Implementation of an Asynchronous Optimization Algorithm for the Parameter Servercited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 16; 2023: 14; 2024: 17; 2025: 15; 2026: 6.

72 papers.

Sort:

By area:

  1. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 79
  2. 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 64
  3. 2021FedLab: A Flexible Federated Learning Frameworkcited by 55
  4. 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
  5. 2021Anarchic Federated Learningcited by 42
  6. 2022Federated Minimax Optimization: Improved Convergence Analyses and Algorithmscited by 37
  7. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 31
  8. 2022Communication-Efficient Device Scheduling for Federated Learning Using Stochastic Optimizationcited by 28
  9. 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
  10. 2022A General Theory for Federated Optimization with Asynchronous and Heterogeneous Clients Updatescited by 17
  11. 2022Semi-Decentralized Federated Learning with Collaborative Relayingcited by 16
  12. 2023AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithmscited by 16
38 more
  1. 2023On the Role of Server Momentum in Federated Learningcited by 16
  2. 2023Accelerating Hybrid Federated Learning Convergence under Partial Participationcited by 15
  3. 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14
  4. 2023Federated Learning with Regularized Client Participationcited by 14
  5. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 13
  6. 2022Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learningcited by 10
  7. 2022FedGS: Federated Graph-based Sampling with Arbitrary Client Availabilitycited by 10
  8. 2023MimiC: Combating Client Dropouts in Federated Learning by Mimicking Central Updatescited by 10
  9. 2023Over-the-Air Federated Learning and Optimizationcited by 10
  10. 2024Accelerating Heterogeneous Federated Learning with Closed-form Classifierscited by 10
  11. 2022Decentralized Event-Triggered Federated Learning with Heterogeneous Communication Thresholdscited by 9
  12. 2024Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailabilitycited by 9
  13. 2021From Deterioration to Acceleration: A Calibration Approach to Rehabilitating Step Asynchronism in Federated Optimizationcited by 8
  14. 2024Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysiscited by 8
  15. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 8
  16. 2024Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clientscited by 7
  17. 2023Towards Bias Correction of FedAvg over Nonuniform and Time-Varying Communicationscited by 6
  18. 2025Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainablecited by 6
  19. 2021Federated Learning with Heterogeneous Differential Privacycited by 5
  20. 2022Robust Federated Learning with Connectivity Failures: A Semi-Decentralized Framework with Collaborative Relayingcited by 5
  21. 2023Federated Learning in the Presence of Adversarial Client Unavailabilitycited by 5
  22. 2024FedStale: leveraging stale client updates in federated learningcited by 5
  23. 2024Riemannian Federated Learning via Averaging Gradient Streamscited by 4
  24. 2022Combating Client Dropout in Federated Learning via Friend Model Substitutioncited by 3
  25. 2022Maximizing Global Model Appeal in Federated Learningcited by 3
  26. 2023Stabilizing and Improving Federated Learning with Non-IID Data and Client Dropoutcited by 3
  27. 2024Empowering Federated Learning with Implicit Gossiping: Mitigating Connection Unreliability Amidst Unknown and Arbitrary Dynamicscited by 3
  28. 2024Understanding Server-Assisted Federated Learning in the Presence of Incomplete Client Participationcited by 3
  29. 2024Federated and Transfer Learning for Cancer Detection Based on Image Analysiscited by 3
  30. 2024Distributed Online Optimization with Stochastic Agent Availabilitycited by 3
  31. 2025Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participationcited by 3
  32. 2022CC-FedAvg: Computationally Customized Federated Averagingcited by 2
  33. 2023Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variancecited by 2
  34. 2023Robust Collaborative Inference with Vertically Split Data Over Dynamic Device Environmentscited by 2
  35. 2024Asynchronous Federated Stochastic Optimization for Heterogeneous Objectives Under Arbitrary Delayscited by 2
  36. 2024Privacy Preserving Semi-Decentralized Mean Estimation over Intermittently-Connected Networkscited by 2
  37. 2023Federated Gradient Matching Pursuitcited by 1
  38. 2023A Lightweight Method for Tackling Unknown Participation Statistics in Federated Averagingcited by 1

50 most cited of 72 All 72 (JSON)