Linearly convergent stochastic heavy ball method for minimizing generalization error
Not on the map. Why?
Abstract (arXiv)
In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder problem. While in the analysis we constrain ourselves to quadratic loss, the overall objective is not necessarily strongly convex.
Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Oct 2017.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 2; 2022: 1; 2023: 6; 2024: 2; 2025: 1; 2026: 2.
14 papers.
Sort:
By area:
- 2021Distributed Second Order Methods with Fast Rates and Compressed Communicationcited by 25
- 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guaranteescited by 22
- 2023Fast Diffusion Modelcited by 21
- 2021AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methodscited by 15
- 2022On the fast convergence of minibatch heavy ball momentumcited by 13
- 2023Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normalitycited by 9
- 2023Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noisecited by 9
- 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 8
- 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
- 2025The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applicationscited by 5
- 2024On the Convergence of FedProx with Extrapolation and Inexact Proxcited by 3
- 2024Parallel Momentum Methods Under Biased Gradient Estimationscited by 2
- 2026Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks5 Jul 2026
- 2026Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration18 May 2026
- 2025The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applications4 Feb 2025
- 2024On the Convergence of FedProx with Extrapolation and Inexact Prox2 Oct 2024
- 2024Parallel Momentum Methods Under Biased Gradient Estimations29 Feb 2024
- 2023Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise22 Dec 2023
- 2023Towards a Better Theoretical Understanding of Independent Subnetwork Training28 Jun 2023
- 2023Fast Diffusion Model12 Jun 2023
- 2023Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normality28 May 2023
- 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees6 May 2023
- 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting7 Feb 2023
- 2022On the fast convergence of minibatch heavy ball momentum15 Jun 2022
- 2023Fast Diffusion Modelcited by 21
- 2023Fast Diffusion Model12 Jun 2023
- 2021Distributed Second Order Methods with Fast Rates and Compressed Communicationcited by 25
- 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guaranteescited by 22
- 2021AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methodscited by 15
- 2022On the fast convergence of minibatch heavy ball momentumcited by 13
- 2023Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normalitycited by 9
- 2023Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noisecited by 9
- 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 8
- 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
- 2025The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applicationscited by 5
- 2024On the Convergence of FedProx with Extrapolation and Inexact Proxcited by 3
- 2024Parallel Momentum Methods Under Biased Gradient Estimationscited by 2
- 2026Unified convergence analysis for gradient descent optimization methods in the training of deep neural networkscited by 1
12 most cited of 13 All 13 (JSON)
- 2026Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks5 Jul 2026
- 2026Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration18 May 2026
- 2025The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applications4 Feb 2025
- 2024On the Convergence of FedProx with Extrapolation and Inexact Prox2 Oct 2024
- 2024Parallel Momentum Methods Under Biased Gradient Estimations29 Feb 2024
- 2023Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise22 Dec 2023
- 2023Towards a Better Theoretical Understanding of Independent Subnetwork Training28 Jun 2023
- 2023Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normality28 May 2023
- 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees6 May 2023
- 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting7 Feb 2023
- 2022On the fast convergence of minibatch heavy ball momentum15 Jun 2022
- 2021AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods19 Feb 2021
12 newest of 13 All 13 (JSON)