mlsanityData to 13 Oct 2026

Stragglers Are Not Disaster: A Hybrid Federated Learning Algorithm with Delayed Gradients

arXiv 2102.06329 ↗First posted 12 Feb 2021cs.LG cs.DCcited by 13

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a new machine learning framework which trains a joint model across a large amount of decentralized computing devices. Existing methods, e.g., Federated Averaging (FedAvg), are able to provide an optimization guarantee by synchronously training the joint model, but usually suffer from stragglers, i.e., IoT devices with low computing power or communication bandwidth, especially on heterogeneous optimization problems. To mitigate the influence of stragglers, this paper presents a novel FL algorithm, namely Hybrid Federated Learning (HFL), to achieve a learning balance in efficiency and effectiveness. It consists of two major components: synchronous kernel and asynchronous updater. Unlike traditional synchronous FL methods, our HFL introduces the asynchronous updater which actively pulls unsynchronized and delayed local weights from stragglers. An adaptive approximation method, Adaptive Delayed-SGD (AD-SGD), is proposed to merge the delayed local updates into the joint model. The theoretical analysis of HFL shows that the convergence rate of the proposed algorithm is O(frac1t+τ) for both convex and non-convex optimization problems.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Builds on 27 of its 41 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

27 of 41 references.

References · 27

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,653
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,328
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,981
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,470
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,227
  6. 2018Federated Learning with Non-IID Datacited by 1,106
  7. 2017Federated Multi-Task Learningcited by 670
  8. 2018Local SGD Converges Fast and Communicates Littlecited by 601
  9. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  10. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 490
  11. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  12. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 409
15 more
  1. 2018Sparsified SGD with Memorycited by 376
  2. 2018Analyzing Federated Learning through an Adversarial Lenscited by 351
  3. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 287
  4. 2014The Loss Surfaces of Multilayer Networkscited by 280
  5. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  6. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  7. 2016Deep Learning without Poor Local Minimacited by 234
  8. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 204
  9. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  10. 2014Recurrent Neural Network Regularizationcited by 189
  11. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 132
  12. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 92
  13. 2018A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updatescited by 47
  14. 2018Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimizationcited by 34
  15. 2020Asynchronous Distributed Optimization with Stochastic Delayscited by 2

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 2; 2023: 3; 2024: 3; 2025: 1.

13 papers.

Sort:

  1. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  2. 2021Non-IID data and Continual Learning processes in Federated Learning: A long road aheadcited by 36
  3. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  4. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
  5. 2021Context-Aware Online Client Selection for Hierarchical Federated Learningcited by 20
  6. 2023FedVS: Straggler-Resilient and Privacy-Preserving Vertical Federated Learning for Split Modelscited by 17
  7. 2022Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropoutcited by 12
  8. 2021FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximationcited by 10
  9. 2022Accelerating Asynchronous Federated Learning Convergence via Opportunistic Mobile Relayingcited by 6
  10. 2024FedCore: Straggler-Free Federated Learning with Distributed Coresetscited by 2
  11. 2024Asynchronous Byzantine Federated Learningcited by 1
  12. 2024DynamicFL: Federated Learning with Dynamic Communication Resource Allocationcited by 1
1 more
  1. 2025Asynchronous Personalized Federated Learning through Global Memorizationcited by 0