# Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data arXiv: https://arxiv.org/abs/2203.01214 ยท HTML: https://mlsanity.com/paper/2203.01214/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 56 references that are in the map. - First posted on arXiv 2 Mar 2022. ## Builds on (most cited first, 30 of 34) - [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019) - [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018) - [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016) - [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019) - [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018) - [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018) - [Federated Machine Learning: Concept and Applications](https://mlsanity.com/paper/1902.04885/) (2019) - [Asynchronous Federated Optimization](https://mlsanity.com/paper/1903.03934/) (2019) - [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015) - [Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization](https://mlsanity.com/paper/1506.08272/) (2015) - [A generic framework for privacy preserving deep learning](https://mlsanity.com/paper/1811.04017/) (2018) - [Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation](https://mlsanity.com/paper/1903.07424/) (2019) - [SAFA: a Semi-Asynchronous Protocol for Fast Federated Learning with Low Overhead](https://mlsanity.com/paper/1910.01355/) (2019) - [First Analysis of Local GD on Heterogeneous Data](https://mlsanity.com/paper/1909.04715/) (2019) - [Variance Reduced Local SGD with Lower Communication Complexity](https://mlsanity.com/paper/1912.12844/) (2019) - [Accelerating Federated Learning via Momentum Gradient Descent](https://mlsanity.com/paper/1910.03197/) (2019) - [Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD](https://mlsanity.com/paper/1803.01113/) (2018) - [MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling](https://mlsanity.com/paper/1905.09435/) (2019) - [Zeno++: Robust Fully Asynchronous SGD](https://mlsanity.com/paper/1903.07020/) (2019) - [Staleness-aware Async-SGD for Distributed Deep Learning](https://mlsanity.com/paper/1511.05950/) (2015) - [Asynchrony begets Momentum, with an Application to Deep Learning](https://mlsanity.com/paper/1605.09774/) (2016) - [Toward Understanding the Impact of Staleness in Distributed Machine Learning](https://mlsanity.com/paper/1810.03264/) (2018) - [Distributed Deep Learning Model for Intelligent Video Surveillance Systems with Edge Computing](https://mlsanity.com/paper/1904.06400/) (2019) - [Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates](https://mlsanity.com/paper/1911.09030/) (2019) - [Towards Efficient Scheduling of Federated Mobile Devices under Computational and Statistical Heterogeneity](https://mlsanity.com/paper/2005.12326/) (2020) - [Asynchronous Federated Learning with Differential Privacy for Edge Intelligence](https://mlsanity.com/paper/1912.07902/) (2019) - [Asynchronous Online Federated Learning for Edge Devices with Non-IID Data](https://mlsanity.com/paper/1911.02134/) (2019) - [On Unbounded Delays in Asynchronous Parallel Fixed-Point Algorithms](https://mlsanity.com/paper/1609.04746/) (2016) ## Built on by (most cited first, 9 of 9) - [FedHiSyn: A Hierarchical Synchronous Federated Learning Framework for Resource and Data Heterogeneity](https://mlsanity.com/paper/2206.10546/) (2022) - [Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions](https://mlsanity.com/paper/2411.12377/) (2024) - [Analysis and Optimization of Wireless Federated Learning with Data Heterogeneity](https://mlsanity.com/paper/2308.03521/) (2023) - [Robust Fully-Asynchronous Methods for Distributed Training over General Architecture](https://mlsanity.com/paper/2307.11617/) (2023) - [Knowledge Rumination for Client Utility Evaluation in Heterogeneous Federated Learning](https://mlsanity.com/paper/2312.10425/) (2023) - [MAB-Based Channel Scheduling for Asynchronous Federated Learning in Non-Stationary Environments](https://mlsanity.com/paper/2503.01324/) (2025) - [Objective Value Change and Shape-Based Accelerated Optimization for the Neural Network Approximation](https://mlsanity.com/paper/2508.20290/) (2025) - [Server-Proximal Aggregation for Federated Domain-Incremental Learning under Partial Participation: Task-Uniform Convergence and Backward Transfer](https://mlsanity.com/paper/2601.22274/) (2026) - [Turning Stale Gradients into Stable Gradients: Coherent Coordinate Descent with Implicit Landscape Smoothing for Lightweight Zeroth-Order Optimization](https://mlsanity.com/paper/2605.14373/) (2026) ## Abstract (arXiv) Federated learning (FL) is an emerging privacy-preserving paradigm that enables multiple participants collaboratively to train a global model without uploading raw data. Considering heterogeneous computing and communication capabilities of different participants, asynchronous FL can avoid the stragglers effect in synchronous FL and adapts to scenarios with vast participants. Both staleness and non-IID data in asynchronous FL would reduce the model utility. However, there exists an inherent contradiction between the solutions to the two problems. That is, mitigating the staleness requires to select less but consistent gradients while coping with non-IID data demands more comprehensive gradients. To address the dilemma, this paper proposes a two-stage weighted K asynchronous FL with adaptive learning rate (WKAFL). By selecting consistent gradients and adjusting learning rate adaptively, WKAFL utilizes stale gradients and mitigates the impact of non-IID data, which can achieve multifaceted enhancement in training speed, prediction accuracy and training stability. We also present the convergence analysis for WKAFL under the assumption of unbounded staleness to understand the impact of staleness and non-IID data. Experiments implemented on both benchmark and synthetic FL datasets show that WKAFL has better overall performance compared to existing algorithms.