mlsanityData to 7 Oct 2026

Toward Understanding the Impact of Staleness in Distributed Machine Learning

arXiv 1810.03264 ↗First posted 8 Oct 2018cs.LG cs.DC stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

Many distributed machine learning (ML) systems adopt the non-synchronous execution in order to alleviate the network communication bottleneck, resulting in stale parameters that do not reflect the latest updates. Despite much development in large-scale ML, the effects of staleness on learning are inconclusive as it is challenging to directly monitor or control staleness in complex distributed environments. In this work, we study the convergence behaviors of a wide array of ML models and algorithms under delayed updates. Our extensive experiments reveal the rich diversity of the effects of staleness on the convergence of ML algorithms and offer insights into seemingly contradictory reports in the literature. The empirical findings also inspire a new convergence analysis of stochastic gradient descent in non-convex optimization under staleness, matching the best-known convergence rate of O(1/√T).

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2020: 1; 2021: 6; 2022: 9; 2023: 5; 2024: 2; 2025: 5.

28 papers.

Sort:

By area:

  1. 2021SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networkscited by 30
  2. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  3. 2024MSPipe: Efficient Temporal GNN Training via Staleness-Aware Pipelinecited by 12
  4. 2021Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settingscited by 11
  5. 2022Convergence Acceleration in Wireless Federated Learning: A Stackelberg Game Approachcited by 10
  6. 2022Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Datacited by 9
  7. 2023Joint Age-based Client Selection and Resource Allocation for Communication-Efficient Federated Learning over NOMA Networkscited by 8
  8. 2023Defending Against Poisoning Attacks in Federated Learning with Blockchaincited by 8
  9. 2022Accelerating Asynchronous Federated Learning Convergence via Opportunistic Mobile Relayingcited by 6
  10. 2025ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updatescited by 5
  11. 2021AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruningcited by 4
  12. 2022FedSS: Federated Learning with Smart Selection of clientscited by 4
16 more
  1. 2024DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learningcited by 4
  2. 2020Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descentcited by 3
  3. 2021Straggler-Resilient Distributed Machine Learning with Dynamic Backup Workerscited by 3
  4. 2022Energy Minimization for Federated Asynchronous Learning on Battery-Powered Mobile Devices via Application Co-runningcited by 3
  5. 2022Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusterscited by 3
  6. 2023Straggler-Resilient Decentralized Learning via Adaptive Asynchronous Updatescited by 3
  7. 2022Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learningcited by 2
  8. 2025Adaptive Deadline and Batch Layered Synchronized Federated Learningcited by 1
  9. 2021Scheduling Optimization Techniques for Neural Network Trainingcited by 0
  10. 2021Wireless-Enabled Asynchronous Federated Fourier Neural Network for Turbulence Prediction in Urban Air Mobility (UAM)cited by 0
  11. 2022Federated Stochastic Gradient Descent Begets Self-Induced Momentumcited by 0
  12. 2022AFAFed -- Protocol analysiscited by 0
  13. 2023Client Orchestration and Cost-Efficient Joint Optimization for NOMA-Enabled Hierarchical Federated Learningcited by 0
  14. 2025Stackelberg Game Based Performance Optimization in Digital Twin Assisted Federated Learning over NOMA Networkscited by 0
  15. 2025Optimizing Value of Learning in Task-Oriented Federated Meta-Learning Systemscited by 0
  16. 2025MAB-Based Channel Scheduling for Asynchronous Federated Learning in Non-Stationary Environmentscited by 0