On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimization
Not on the map. Why?
Abstract (arXiv)
Despite their popularity, the practical performance of asynchronous stochastic gradient descent methods (ASGD) for solving large scale machine learning problems are not as good as theoretical results indicate. We adopt and analyze a synchronous K-step averaging stochastic gradient descent algorithm which we call K-AVG. We establish the convergence results of K-AVG for nonconvex objectives and explain why the K-step delay is necessary and leads to better performance than traditional parallel stochastic gradient descent which is a special case of K-AVG with K=1. We also show that K-AVG scales better than ASGD. Another advantage of K-AVG over ASGD is that it allows larger stepsizes. On a cluster of 128 GPUs, K-AVG is faster than ASGD implementations and achieves better accuracies and faster convergence for cifar dataset.
Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Aug 2017.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 34; 2022: 22; 2023: 12; 2024: 13; 2025: 6; 2026: 4.
91 papers.
Sort:
- 2021Personalized Federated Learning using Hypernetworkscited by 210
- 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
- 2021Personalized Federated Learning with Gaussian Processescited by 76
- 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
- 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networkscited by 64
- 2021What Do We Mean by Generalization in Federated Learning?cited by 62
- 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
- 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
- 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
- 2021A Bayesian Federated Learning Framework with Online Laplace Approximationcited by 41
- 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
38 more
- 2024Hierarchical Split Federated Learning: Convergence Analysis and System Optimizationcited by 38
- 2022Federated Minimax Optimization: Improved Convergence Analyses and Algorithmscited by 37
- 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
- 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 36
- 2021Efficient Device Scheduling with Multi-Job Federated Learningcited by 29
- 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
- 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
- 2022Multi-Job Intelligent Scheduling with Cross-Device Federated Learningcited by 20
- 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
- 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 19
- 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19
- 2021Fast Federated Learning by Balancing Communication Trade-Offscited by 18
- 2021On Bridging Generic and Personalized Federated Learning for Image Classificationcited by 15
- 2025HASFL: Heterogeneity-aware Split Federated Learning over Edge Computing Systemscited by 15
- 2021Stragglers Are Not Disaster: A Hybrid Federated Learning Algorithm with Delayed Gradientscited by 13
- 2023Recent Methodological Advances in Federated Learning for Healthcarecited by 11
- 2021Wirelessly Powered Federated Edge Learning: Optimal Tradeoffs Between Convergence and Power Transfercited by 10
- 2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10
- 2021FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximationcited by 10
- 2023Over-the-Air Federated Learning and Optimizationcited by 10
- 2021Federated Learning with Unreliable Clients: Performance Analysis and Mechanism Designcited by 9
- 2022When Do Curricula Work in Federated Learning?cited by 9
- 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
- 2021Federated Learning over Noisy Channels: Convergence Analysis and Design Examplescited by 8
- 2021From Deterioration to Acceleration: A Calibration Approach to Rehabilitating Step Asynchronism in Federated Optimizationcited by 8
- 2022FedGiA: An Efficient Hybrid Algorithm for Federated Learningcited by 8
- 2022Fast Heterogeneous Federated Learning with Hybrid Client Selectioncited by 8
- 2023Making Batch Normalization Great in Federated Deep Learningcited by 8
- 2023Mean Estimation with User-level Privacy under Data Heterogeneitycited by 8
- 2024Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clientscited by 7
- 2021CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learningcited by 6
- 2021The Minimax Complexity of Distributed Optimizationcited by 6
- 2021A Stochastic Newton Algorithm for Distributed Convex Optimizationcited by 6
- 2022Personalized Federated Learning with Communication Compressioncited by 6
- 2022Fast-Convergent Federated Learning via Cyclic Aggregationcited by 6
- 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Trainingcited by 6
- 2024Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Datacited by 6
- 2021The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communicationcited by 5
50 most cited of 91 All 91 (JSON)
- 2026Fine-Tuning of Transformer models with Frames26 Aug 2026
- 2026Optimizing the Preconditioner: A Black-box Online-to-Nonconvex Conversion with Static Regret Minimization Oracles20 Jul 2026
- 2026Using Laplace Transform To Optimize the Hallucination of Generation Models5 Mar 2026
- 2026Variance Reduction Based Experience Replay for Policy Optimization5 Feb 2026
- 2025When to Stop Federated Learning: Zero-Shot Generation of Synthetic Validation Data with Generative AI for Early Stopping14 Nov 2025
- 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning27 Jul 2025
- 2025Optimal Batch-Size Control for Low-Latency Federated Learning with Device Heterogeneity21 Jul 2025
- 2025FedOne: Query-Efficient Federated Learning for Black-box Discrete Prompt Learning17 Jun 2025
- 2025HASFL: Heterogeneity-aware Split Federated Learning over Edge Computing Systems10 Jun 2025
- 2025Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysis8 Jan 2025
- 2024Hierarchical Split Federated Learning: Convergence Analysis and System Optimization10 Dec 2024
- 2024Generative AI-Powered Plugin for Robust Federated Learning in Heterogeneous IoT Networks31 Oct 2024
38 more
- 2024Riemannian Federated Learning via Averaging Gradient Streams11 Sep 2024
- 2024Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Data11 Aug 2024
- 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods20 Jun 2024
- 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training3 Jun 2024
- 2024Communication-Efficient Distributed Deep Learning via Federated Dynamic Averaging31 May 2024
- 2024PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory System10 Apr 2024
- 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey9 Apr 2024
- 2024Approximate Gradient Coding for Privacy-Flexible Federated Learning with Non-IID Data4 Apr 2024
- 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networks19 Mar 2024
- 2024Improving LoRA in Privacy-preserving Federated Learning18 Mar 2024
- 2024Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clients17 Feb 2024
- 2023A Quadratic Synchronization Rule for Distributed Deep Learning22 Oct 2023
- 2023Over-the-Air Federated Learning and Optimization16 Oct 2023
- 2023Based on What We Can Control Artificial Neural Networks9 Oct 2023
- 2023Recent Methodological Advances in Federated Learning for Healthcare4 Oct 2023
- 2023Mitigating Group Bias in Federated Learning for Heterogeneous Devices13 Sep 2023
- 2023Federated Learning Robust to Byzantine Attacks: Achieving Zero Optimality Gap21 Aug 2023
- 2023Mean Estimation with User-level Privacy under Data Heterogeneity28 Jul 2023
- 2023Momentum Benefits Non-IID Federated Learning Simply and Provably28 Jun 2023
- 2023Federated Composite Saddle Point Optimization25 May 2023
- 2023Asynchronous Multi-Model Dynamic Federated Learning over Wireless Networks: Theory, Modeling, and Optimization22 May 2023
- 2023Hierarchical Weight Averaging for Deep Neural Networks23 Apr 2023
- 2023Making Batch Normalization Great in Federated Deep Learning12 Mar 2023
- 2022When Do Curricula Work in Federated Learning?24 Dec 2022
- 2022Multi-Job Intelligent Scheduling with Cross-Device Federated Learning24 Nov 2022
- 2022Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimization10 Nov 2022
- 2022Fast-Convergent Federated Learning via Cyclic Aggregation29 Oct 2022
- 2022STSyn: Speeding Up Local SGD with Straggler-Tolerant Synchronization6 Oct 2022
- 2022Scaling up Stochastic Gradient Descent for Non-convex Optimisation6 Oct 2022
- 2022SAGDA: Achieving O(ε-2) Communication Complexity in Federated Min-Max Learning2 Oct 2022
- 2022Personalized Federated Learning with Communication Compression12 Sep 2022
- 2022Variance Reduction based Experience Replay for Policy Optimization25 Aug 2022
- 2022Fast Heterogeneous Federated Learning with Hybrid Client Selection10 Aug 2022
- 2022Multi-Model Federated Learning with Provable Guarantees9 Jul 2022
- 2022On the Importance and Applicability of Pre-Training for Federated Learning23 Jun 2022
- 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data9 Jun 2022
- 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learning27 May 2022
- 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks10 May 2022
50 newest of 91 All 91 (JSON)
- 2021Personalized Federated Learning using Hypernetworkscited by 210
- 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
- 2021Personalized Federated Learning with Gaussian Processescited by 76
- 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
- 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networkscited by 64
- 2021What Do We Mean by Generalization in Federated Learning?cited by 62
- 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
- 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
- 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
- 2021A Bayesian Federated Learning Framework with Online Laplace Approximationcited by 41
- 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
12 most cited of 91 All 91 (JSON)
- 2026Fine-Tuning of Transformer models with Frames26 Aug 2026
- 2026Optimizing the Preconditioner: A Black-box Online-to-Nonconvex Conversion with Static Regret Minimization Oracles20 Jul 2026
- 2026Using Laplace Transform To Optimize the Hallucination of Generation Models5 Mar 2026
- 2026Variance Reduction Based Experience Replay for Policy Optimization5 Feb 2026
- 2025When to Stop Federated Learning: Zero-Shot Generation of Synthetic Validation Data with Generative AI for Early Stopping14 Nov 2025
- 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning27 Jul 2025
- 2025Optimal Batch-Size Control for Low-Latency Federated Learning with Device Heterogeneity21 Jul 2025
- 2025FedOne: Query-Efficient Federated Learning for Black-box Discrete Prompt Learning17 Jun 2025
- 2025HASFL: Heterogeneity-aware Split Federated Learning over Edge Computing Systems10 Jun 2025
- 2025Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysis8 Jan 2025
- 2024Hierarchical Split Federated Learning: Convergence Analysis and System Optimization10 Dec 2024
- 2024Generative AI-Powered Plugin for Robust Federated Learning in Heterogeneous IoT Networks31 Oct 2024
12 newest of 91 All 91 (JSON)