mlsanityData to 7 Oct 2026

Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentum

arXiv 2111.06171 ↗First posted 11 Nov 2021math.OC cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient descent with momentum (SGDM) is the dominant algorithm in many optimization scenarios, including convex optimization instances and non-convex neural network training. Yet, in the stochastic setting, momentum interferes with gradient noise, often leading to specific step size and momentum choices in order to guarantee convergence, set aside acceleration. Proximal point methods, on the other hand, have gained much attention due to their numerical stability and elasticity against imperfect tuning. Their stochastic accelerated variants though have received limited attention: how momentum interacts with the stability of (stochastic) proximal point methods remains largely unstudied. To address this, we focus on the convergence and stability of the stochastic proximal point algorithm with momentum (SPPAM), and show that SPPAM allows a faster linear convergence to a neighborhood compared to the stochastic proximal point algorithm (SPPA) with a better contraction factor, under proper hyperparameter tuning. In terms of stability, we show that SPPAM depends on problem constants more favorably than SGDM, allowing a wider range of step size and momentum that lead to convergence.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 26 of its 55 references that are in the map. First posted on arXiv 11 Nov 2021.

Builds on

26 of 55 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  5. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  6. 2019SGD: General Analysis and Improved Ratescited by 206
  7. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  8. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 110
  9. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 108
  10. 2014Analysis and Design of Optimization Algorithms via Integral Quadratic Constraintscited by 105
  11. 2018Stochastic (Approximate) Proximal Point Methods: Convergence, Optimality, and Adaptivitycited by 61
  12. 2014Linear Coupling: An Ultimate Unification of Gradient and Mirror Descentcited by 58
14 more
  1. 2015A Universal Catalyst for First-Order Optimizationcited by 54
  2. 2017Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methodscited by 49
  3. 2018On the insufficiency of existing momentum schemes for Stochastic Optimizationcited by 46
  4. 2017Catalyst Acceleration for First-order Convex Optimization: from Theory to Practicecited by 39
  5. 2020On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settingscited by 27
  6. 2017Dissipativity Theory for Nesterov's Accelerated Methodcited by 22
  7. 2015A geometric alternative to Nesterov's accelerated gradient descentcited by 21
  8. 2015The Proximal Robbins-Monro Methodcited by 14
  9. 2021Accelerated, Optimal, and Parallel: Some Results on Model-Based Stochastic Optimizationcited by 14
  10. 2019Nesterov's method with decreasing learning rate leads to accelerated stochastic gradient descentcited by 13
  11. 2021Minibatch and Momentum Model-based Methods for Stochastic Weakly Convex Optimizationcited by 10
  12. 2019A Generic Acceleration Framework for Stochastic Composite Optimizationcited by 9
  13. 2020Understanding Nesterov's Acceleration via Proximal Point Methodcited by 7
  14. 2018On the Curved Geometry of Accelerated Optimizationcited by 6

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 1.

7 papers.

  1. 2022Faster federated optimization under second-order similaritycited by 18
  2. 2023Adaptive Federated Learning with Auto-Tuned Clientscited by 7
  3. 2024A Unified Theory of Stochastic Proximal Point Methods without Smoothnesscited by 4
  4. 2023Variance reduction techniques for stochastic proximal point algorithmscited by 3
  5. 2023On First-Order Meta-Reinforcement Learning with Moreau Envelopescited by 1
  6. 2025Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimizationcited by 1
  7. 2024A Catalyst Framework for the Quantum Linear System Problem via the Proximal Point Algorithmcited by 0