Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization
Not on the map. Why?
Abstract (arXiv)
The present article studies the minimization of convex, L-smooth functions defined on a separable real Hilbert space. We analyze regularized stochastic gradient descent (reg-SGD), a variant of stochastic gradient descent that uses a Tikhonov regularization with time-dependent, vanishing regularization parameter. We prove strong convergence of reg-SGD to the minimum-norm solution of the original problem without additional boundedness assumptions. Moreover, we quantify the rate of convergence and optimize the interplay between step-sizes and regularization decay. Our analysis reveals how vanishing Tikhonov regularization controls the flow of SGD and yields stable learning dynamics, offering new insights into the design of iterative algorithms for convex problems, including those that arise in ill-posed inverse problems. We validate our theoretical findings through numerical experiments on image reconstruction and ODE-based inverse problems.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 25 of its 73 references that are in the map. First posted on arXiv 16 May 2025.
Builds on
25 of 73 references.
References · 25
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
- 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
- 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
- 2020Better Theory for SGD in the Nonconvex Worldcited by 158
- 2020Implicit Gradient Regularizationcited by 146
- 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
- 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
- 2020SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolationcited by 67
- 2015From error bounds to the complexity of first-order descent methods for convex functionscited by 43
- 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 39
- 2023Revisiting the Last-Iterate Convergence of Stochastic Gradient Methodscited by 38
13 more
- 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 28
- 2022Sharp Analysis of Stochastic Optimization under Global Kurdyka-Lojasiewicz Inequalitycited by 20
- 2017Convergence of the Forward-Backward Algorithm: Beyond the Worst Case with the Help of Geometrycited by 7
- 2020Explicit Regularization of Stochastic Gradient Methods through Dualitycited by 6
- 2022Statistical Learning and Inverse Problems: A Stochastic Gradient Approachcited by 5
- 2019On the Convergence of Stochastic Gradient Descent for Nonlinear Ill-Posed Problemscited by 4
- 2020On the Saturation Phenomenon of Stochastic Gradient Descent for Linear Inverse Problemscited by 3
- 2024A Guide to Stochastic Optimisation for Large-Scale Inverse Problemscited by 3
- 2016Combining fast inertial dynamics for convex optimization with Tikhonov regularizationcited by 2
- 2021Convex optimization via inertial algorithms with vanishing Tikhonov regularization: fast convergence to the minimum norm solutioncited by 2
- 2014Splitting forward-backward penalty scheme for constrained variational problemscited by 1
- 2022Accelerated gradient methods with strong convergence to the minimum norm minimizer: a dynamic approach combining time scaling, averaging, and Tikhonov regularizationcited by 1
- 2024Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spacescited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.