Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
Not on the map. Why?
Abstract (arXiv)
Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies remains a largely open problem. This paper presents a unified framework called Cooperative SGD that subsumes existing communication-efficient SGD algorithms such as periodic-averaging, elastic-averaging and decentralized SGD. By analyzing Cooperative SGD, we provide novel convergence guarantees for existing algorithms. Moreover, this framework enables us to design new communication-efficient SGD algorithms that strike the best balance between reducing communication overhead and achieving fast error convergence with low error floor.
Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Aug 2018.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2020: 1; 2021: 57; 2022: 27; 2023: 10; 2024: 6; 2025: 1; 2026: 4.
106 papers.
Sort:
By area:
- 2021A Field Guide to Federated Optimizationcited by 315
- 2021Personalized Federated Learning using Hypernetworkscited by 212
- 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
- 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
- 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 97
- 2021CRFL: Certifiably Robust Federated Learning against Backdoor Attackscited by 91
- 2021Personalized Federated Learning with Gaussian Processescited by 76
- 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
- 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 64
- 2021What Do We Mean by Generalization in Federated Learning?cited by 62
- 2021Federated Learning with Fair Averagingcited by 61
- 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
38 more
- 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 56
- 2021Decentralized Federated Learning: Balancing Communication and Computing Costscited by 55
- 2021Deep Federated Learning for Autonomous Drivingcited by 54
- 2021Hierarchical Federated Learning with Quantization: Convergence Analysis and System Designcited by 47
- 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
- 2021Anarchic Federated Learningcited by 42
- 2021Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiencycited by 40
- 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
- 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 38
- 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 36
- 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participationcited by 36
- 2021Mobility-Aware Cluster Federated Learning in Hierarchical Wireless Networkscited by 35
- 2021The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communicationcited by 31
- 2021SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networkscited by 30
- 2021FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimizationcited by 29
- 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 27
- 2021Decentralized Local Stochastic Extra-Gradient for Variational Inequalitiescited by 25
- 2021Cost-Effective Federated Learning in Mobile Edge Networkscited by 25
- 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
- 2021Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfercited by 23
- 2022Gradient Masked Averaging for Federated Learningcited by 22
- 2021Federated Learning with Taskonomy for Non-IID Datacited by 21
- 2022SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communicationcited by 21
- 2021Exploiting Heterogeneity in Robust Federated Best-Arm Identificationcited by 20
- 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 20
- 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
- 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
- 2021Fast Federated Learning by Balancing Communication Trade-Offscited by 18
- 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
- 2021Joint Optimization of Communications and Federated Learning Over the Aircited by 14
- 2023Communication-Efficient Federated Hypergradient Computation via Aggregated Iterative Differentiationcited by 13
- 2023Guiding The Last Layer in Federated Learning with Pre-Trained Modelscited by 13
- 2021Auto-weighted Robust Federated Learning with Corrupted Data Sourcescited by 12
- 2022Data-heterogeneity-aware Mixing for Decentralized Learningcited by 12
- 2021Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settingscited by 11
- 20211-Bit Compressive Sensing for Efficient Federated Learning Over the Aircited by 11
- 2021FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learningcited by 11
- 2021Handling Data Heterogeneity with Generative Replay in Collaborative Learning for Medical Imagingcited by 9
50 most cited of 106 All 106 (JSON)
- 2026LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging20 May 2026
- 2026Event-Triggered Gossip for Distributed Learning22 Feb 2026
- 2026Tight Analysis of Decentralized SGD: A Markov Chain Perspective11 Jan 2026
- 2026Accelerating Decentralized Optimization via Overlapping Local Steps4 Jan 2026
- 2025Lightweight Temporal Transformer Decomposition for Federated Autonomous Driving30 Jun 2025
- 2024Review of Mathematical Optimization in Federated Learning2 Dec 2024
- 2024Refined Analysis of Federated Averaging and Federated Richardson-Romberg2 Dec 2024
- 2024Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysis30 Oct 2024
- 2024Federated Learning with Integrated Sensing, Communication, and Computation: Frameworks and Performance Analysis17 Sep 2024
- 2024A New Theoretical Perspective on Data Heterogeneity in Federated Optimization22 Jul 2024
- 2024Differentiated Aggregation to Improve Generalization in Federated Learning17 Apr 2024
- 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction12 Oct 2023
38 more
- 2023DRAG: Divergence-based Adaptive Aggregation in Federated learning on Non-IID Data4 Sep 2023
- 2023Guiding The Last Layer in Federated Learning with Pre-Trained Models6 Jun 2023
- 2023Federated Multi-Sequence Stochastic Approximation with Local Hypergradient Estimation2 Jun 2023
- 2023Theoretically Principled Federated Learning for Balancing Privacy and Utility24 May 2023
- 2023Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning11 Apr 2023
- 2023On the Local Cache Update Rules in Streaming Federated Learning28 Mar 2023
- 2023EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Data14 Feb 2023
- 2023Communication-Efficient Federated Hypergradient Computation via Aggregated Iterative Differentiation9 Feb 2023
- 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneity4 Feb 2023
- 2022SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communication25 Oct 2022
- 2022Latency Aware Semi-synchronous Client Selection and Model Aggregation for Wireless Federated Learning19 Oct 2022
- 2022Random Orthogonalization for Federated Learning in Massive MIMO Systems18 Oct 2022
- 2022STSyn: Speeding Up Local SGD with Straggler-Tolerant Synchronization6 Oct 2022
- 2022SAGDA: Achieving O(ε-2) Communication Complexity in Federated Min-Max Learning2 Oct 2022
- 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learning28 Jul 2022
- 2022Reducing Training Time in Cross-Silo Federated Learning using Multigraph Topology20 Jul 2022
- 2022Multi-Model Federated Learning with Provable Guarantees9 Jul 2022
- 2022Fundamental Limits of Communication Efficiency for Model Aggregation in Distributed Learning: A Rate-Distortion Approach28 Jun 2022
- 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data9 Jun 2022
- 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learning27 May 2022
- 2022Combating Client Dropout in Federated Learning via Friend Model Substitution26 May 2022
- 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks10 May 2022
- 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimization4 May 2022
- 2022Data-heterogeneity-aware Mixing for Decentralized Learning13 Apr 2022
- 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participation28 Mar 2022
- 2022Sparse Federated Learning with Hierarchical Personalized Models25 Mar 2022
- 2022Local Stochastic Factored Gradient Descent for Distributed Quantum State Tomography22 Mar 2022
- 2022The Role of Local Steps in Local SGD14 Mar 2022
- 2022Update Compression for Deep Neural Networks on the Edge9 Mar 2022
- 2022On Federated Learning with Energy Harvesting Clients12 Feb 2022
- 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?1 Feb 2022
- 2022Communication-Efficient Consensus Mechanism for Federated Reinforcement Learning30 Jan 2022
- 2022Gradient Masked Averaging for Federated Learning28 Jan 2022
- 2022On the Convergence of Heterogeneous Federated Learning with Arbitrary Adaptive Online Model Pruning27 Jan 2022
- 2022Partial Model Averaging in Federated Learning: Performance Guarantees and Benefits11 Jan 2022
- 2022Federated Optimization of Smooth Loss Functions6 Jan 2022
- 2021Communication-Efficient Distributed SGD with Compressed Sensing15 Dec 2021
- 2021Efficient and Reliable Overlay Networks for Decentralized Federated Learning12 Dec 2021
50 newest of 106 All 106 (JSON)
- 2021Deep Federated Learning for Autonomous Drivingcited by 54
- 2021Deep Federated Learning for Autonomous Driving12 Oct 2021
- 2021A Field Guide to Federated Optimizationcited by 315
- 2021Personalized Federated Learning using Hypernetworkscited by 212
- 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
- 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
- 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 97
- 2021CRFL: Certifiably Robust Federated Learning against Backdoor Attackscited by 91
- 2021Personalized Federated Learning with Gaussian Processescited by 76
- 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
- 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 64
- 2021What Do We Mean by Generalization in Federated Learning?cited by 62
- 2021Federated Learning with Fair Averagingcited by 61
- 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
12 most cited of 104 All 104 (JSON)
- 2026LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging20 May 2026
- 2026Event-Triggered Gossip for Distributed Learning22 Feb 2026
- 2026Tight Analysis of Decentralized SGD: A Markov Chain Perspective11 Jan 2026
- 2026Accelerating Decentralized Optimization via Overlapping Local Steps4 Jan 2026
- 2025Lightweight Temporal Transformer Decomposition for Federated Autonomous Driving30 Jun 2025
- 2024Review of Mathematical Optimization in Federated Learning2 Dec 2024
- 2024Refined Analysis of Federated Averaging and Federated Richardson-Romberg2 Dec 2024
- 2024Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysis30 Oct 2024
- 2024Federated Learning with Integrated Sensing, Communication, and Computation: Frameworks and Performance Analysis17 Sep 2024
- 2024A New Theoretical Perspective on Data Heterogeneity in Federated Optimization22 Jul 2024
- 2024Differentiated Aggregation to Improve Generalization in Federated Learning17 Apr 2024
- 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction12 Oct 2023
12 newest of 104 All 104 (JSON)