SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learning
Not on the map. Why?
Abstract (arXiv)
Cross-device training is a crucial subfield of federated learning, where the number of clients can reach into the billions. Standard approaches and local methods are prone to issues such as client drift and insensitivity to data similarities. We propose a novel algorithm (SPAM) for cross-device federated learning with non-convex losses, which solves both issues. We provide sharp analysis under second-order (Hessian) similarity, a condition satisfied by a variety of machine learning problems in practice. Additionally, we extend our results to the partial participation setting, where a cohort of selected clients communicate with the server at each communication round. Our method is the first in its kind, that does not require the smoothness of the objective and provably benefits from clients having similar data.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 30 May 2024.
Builds on
39 of 52 references.
References · 39
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
- 2020Adaptive Federated Optimizationcited by 1,036
- 2021Federated Learning Based on Dynamic Regularizationcited by 555
- 2019Federated Learning with Personalization Layerscited by 533
- 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
- 2021A Field Guide to Federated Optimizationcited by 312
- 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 303
- 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
- 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
27 more
- 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
- 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
- 2020On Biased Compression for Distributed Learningcited by 140
- 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
- 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 131
- 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 121
- 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 116
- 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
- 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
- 2019Adaptive Gradient Descent without Descentcited by 79
- 2020Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressorcited by 45
- 2022On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyondcited by 37
- 2014Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learningcited by 35
- 2021Permutation Compressors for Provably Faster Distributed Nonconvex Optimizationcited by 32
- 2020Statistically Preconditioned Accelerated Gradient Method for Distributed Optimizationcited by 29
- 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19
- 2022Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Proxcited by 19
- 2022Faster federated optimization under second-order similaritycited by 18
- 2024Directional Smoothness and Gradient Methods: Convergence and Adaptivitycited by 15
- 2018Optimizing the Efficiency of First-Order Methods for Decreasing the Gradient of Smooth Convex Functionscited by 14
- 2018Primal-dual accelerated gradient methods with small-dimensional relaxation oraclecited by 14
- 2022Optimization with Access to Auxiliary Informationcited by 12
- 2023Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysiscited by 11
- 2022Sharper Rates and Flexible Framework for Nonconvex SGD with Client and Data Samplingcited by 9
- 2023Two Losses Are Better Than One: Faster Optimization Using a Cheaper Proxycited by 7
- 2022Shifted Compression Framework: Generalizations and Improvementscited by 6
- 2022FedEmbed: Personalized Private Federated Learningcited by 3
Built on by
Citing papers per year: 2024: 3.
3 papers.