Generalization Properties and Implicit Regularization for Multiple Passes SGM
Not on the map. Why?
Abstract (arXiv)
We study the generalization properties of stochastic gradient methods for learning with convex loss functions and linearly parameterized functions. We show that, in the absence of penalizations or constraints, the stability and approximation properties of the algorithm can be controlled by tuning either the step-size or the number of passes over the data. In this view, these parameters can be seen to control a form of implicit regularization. Numerical results complement the theoretical findings.
Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 May 2016.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 5; 2022: 5; 2023: 5; 2024: 1.
16 papers.
Sort:
- 2022Overview frequency principle/spectral bias in deep learningcited by 68
- 2021Stability and Generalization of Stochastic Gradient Methods for Minimax Problemscited by 35
- 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problemscited by 25
- 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
- 2021Learning with Gradient Descent and Weakly Convex Lossescited by 16
- 2023Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithmcited by 10
- 2022High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noisecited by 9
- 2021From inexact optimization to learning via gradient concentrationcited by 4
- 2023Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsizecited by 2
- 2023On the Stability and Generalization of Triplet Learningcited by 2
- 2023Minimax Excess Risk of First-Order Methods for Statistical Learning with Data-Dependent Oraclescited by 2
- 2023Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimizationcited by 2
4 more
- 2021Why Does Multi-Epoch Training Help?cited by 1
- 2024Convex SGD: Generalization Without Early Stoppingcited by 1
- 2021Stability and Generalization of the Decentralized Stochastic Gradient Descentcited by 0
- 2022Local SGD in Overparameterized Linear Regressioncited by 0
- 2024Convex SGD: Generalization Without Early Stopping8 Jan 2024
- 2023Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization15 Nov 2023
- 2023Minimax Excess Risk of First-Order Methods for Statistical Learning with Data-Dependent Oracles10 Jul 2023
- 2023Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithm5 Jun 2023
- 2023On the Stability and Generalization of Triplet Learning20 Feb 2023
- 2023Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize10 Feb 2023
- 2022Local SGD in Overparameterized Linear Regression20 Oct 2022
- 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks19 Sep 2022
- 2022High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noise17 Aug 2022
- 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problems14 Jun 2022
- 2022Overview frequency principle/spectral bias in deep learning19 Jan 2022
- 2021From inexact optimization to learning via gradient concentration9 Jun 2021
4 more
- 2021Why Does Multi-Epoch Training Help?13 May 2021
- 2021Stability and Generalization of Stochastic Gradient Methods for Minimax Problems8 May 2021
- 2021Stability and Generalization of the Decentralized Stochastic Gradient Descent2 Feb 2021
- 2021Learning with Gradient Descent and Weakly Convex Losses13 Jan 2021
- 2022Overview frequency principle/spectral bias in deep learningcited by 68
- 2021Stability and Generalization of Stochastic Gradient Methods for Minimax Problemscited by 35
- 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problemscited by 25
- 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
- 2021Learning with Gradient Descent and Weakly Convex Lossescited by 16
- 2023Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithmcited by 10
- 2022High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noisecited by 9
- 2021From inexact optimization to learning via gradient concentrationcited by 4
- 2023Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsizecited by 2
- 2023On the Stability and Generalization of Triplet Learningcited by 2
- 2023Minimax Excess Risk of First-Order Methods for Statistical Learning with Data-Dependent Oraclescited by 2
- 2023Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimizationcited by 2
12 most cited of 16 All 16 (JSON)
- 2024Convex SGD: Generalization Without Early Stopping8 Jan 2024
- 2023Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization15 Nov 2023
- 2023Minimax Excess Risk of First-Order Methods for Statistical Learning with Data-Dependent Oracles10 Jul 2023
- 2023Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithm5 Jun 2023
- 2023On the Stability and Generalization of Triplet Learning20 Feb 2023
- 2023Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize10 Feb 2023
- 2022Local SGD in Overparameterized Linear Regression20 Oct 2022
- 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks19 Sep 2022
- 2022High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noise17 Aug 2022
- 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problems14 Jun 2022
- 2022Overview frequency principle/spectral bias in deep learning19 Jan 2022
- 2021From inexact optimization to learning via gradient concentration9 Jun 2021
12 newest of 16 All 16 (JSON)