mlsanityData to 7 Oct 2026

Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data

arXiv 2203.01214 ↗First posted 2 Mar 2022cs.LG cs.DCcited by 9

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is an emerging privacy-preserving paradigm that enables multiple participants collaboratively to train a global model without uploading raw data. Considering heterogeneous computing and communication capabilities of different participants, asynchronous FL can avoid the stragglers effect in synchronous FL and adapts to scenarios with vast participants. Both staleness and non-IID data in asynchronous FL would reduce the model utility. However, there exists an inherent contradiction between the solutions to the two problems. That is, mitigating the staleness requires to select less but consistent gradients while coping with non-IID data demands more comprehensive gradients. To address the dilemma, this paper proposes a two-stage weighted K asynchronous FL with adaptive learning rate (WKAFL). By selecting consistent gradients and adjusting learning rate adaptively, WKAFL utilizes stale gradients and mitigates the impact of non-IID data, which can achieve multifaceted enhancement in training speed, prediction accuracy and training stability. We also present the convergence analysis for WKAFL under the assumption of unbounded staleness to understand the impact of staleness and non-IID data. Experiments implemented on both benchmark and synthetic FL datasets show that WKAFL has better overall performance compared to existing algorithms.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 2 Mar 2022.

Builds on

34 of 56 references.

References · 34

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  7. 2018Federated Learning with Non-IID Datacited by 1,099
  8. 2018LEAF: A Benchmark for Federated Settingscited by 780
  9. 2019Federated Machine Learning: Concept and Applicationscited by 409
  10. 2019Asynchronous Federated Optimizationcited by 264
  11. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  12. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
22 more
  1. 2018A generic framework for privacy preserving deep learningcited by 103
  2. 2019Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregationcited by 99
  3. 2019SAFA: a Semi-Asynchronous Protocol for Fast Federated Learning with Low Overheadcited by 82
  4. 2019First Analysis of Local GD on Heterogeneous Datacited by 80
  5. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  6. 2019Accelerating Federated Learning via Momentum Gradient Descentcited by 75
  7. 2018Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGDcited by 68
  8. 2019MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Samplingcited by 67
  9. 2019Zeno++: Robust Fully Asynchronous SGDcited by 53
  10. 2015Staleness-aware Async-SGD for Distributed Deep Learningcited by 43
  11. 2016Asynchrony begets Momentum, with an Application to Deep Learningcited by 30
  12. 2018Toward Understanding the Impact of Staleness in Distributed Machine Learningcited by 28
  13. 2019Distributed Deep Learning Model for Intelligent Video Surveillance Systems with Edge Computingcited by 28
  14. 2019Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Ratescited by 26
  15. 2020Towards Efficient Scheduling of Federated Mobile Devices under Computational and Statistical Heterogeneitycited by 20
  16. 2019Asynchronous Federated Learning with Differential Privacy for Edge Intelligencecited by 16
  17. 2019Asynchronous Online Federated Learning for Edge Devices with Non-IID Datacited by 9
  18. 2016On Unbounded Delays in Asynchronous Parallel Fixed-Point Algorithmscited by 7
  19. 2018A Bi-layered Parallel Training Architecture for Large-scale Convolutional Neural Networkscited by 6
  20. 2017More Iterations per Second, Same Quality -- Why Asynchronous Algorithms may Drastically Outperform Traditional Onescited by 5
  21. 2019Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Trainingcited by 5
  22. 2020Privacy-Preserving News Recommendation Model Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 1; 2025: 2; 2026: 2.

9 papers.

  1. 2022FedHiSyn: A Hierarchical Synchronous Federated Learning Framework for Resource and Data Heterogeneitycited by 20
  2. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 12
  3. 2023Analysis and Optimization of Wireless Federated Learning with Data Heterogeneitycited by 3
  4. 2023Robust Fully-Asynchronous Methods for Distributed Training over General Architecturecited by 2
  5. 2023Knowledge Rumination for Client Utility Evaluation in Heterogeneous Federated Learningcited by 0
  6. 2025MAB-Based Channel Scheduling for Asynchronous Federated Learning in Non-Stationary Environmentscited by 0
  7. 2025Objective Value Change and Shape-Based Accelerated Optimization for the Neural Network Approximationcited by 0
  8. 2026Server-Proximal Aggregation for Federated Domain-Incremental Learning under Partial Participation: Task-Uniform Convergence and Backward Transfercited by 0
  9. 2026Turning Stale Gradients into Stable Gradients: Coherent Coordinate Descent with Implicit Landscape Smoothing for Lightweight Zeroth-Order Optimizationcited by 0