# Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization

arXiv: https://arxiv.org/abs/2505.11434 · HTML: https://mlsanity.com/paper/2505.11434/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 73 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [On the Origin of Implicit Regularization in Stochastic Gradient Descent](https://mlsanity.com/paper/2101.12176/) (2021)
- [Handbook of Convergence Theorems for (Stochastic) Gradient Methods](https://mlsanity.com/paper/2301.11235/) (2023)
- [Better Theory for SGD in the Nonconvex World](https://mlsanity.com/paper/2002.03329/) (2020)
- [Implicit Gradient Regularization](https://mlsanity.com/paper/2009.11162/) (2020)
- [On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems](https://mlsanity.com/paper/2006.11144/) (2020)
- [Why Do We Need Weight Decay in Modern Deep Learning?](https://mlsanity.com/paper/2310.04415/) (2023)
- [SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation](https://mlsanity.com/paper/2006.10311/) (2020)
- [From error bounds to the complexity of first-order descent methods for convex functions](https://mlsanity.com/paper/1510.08234/) (2015)
- [Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis](https://mlsanity.com/paper/2105.01650/) (2021)
- [Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods](https://mlsanity.com/paper/2312.08531/) (2023)
- [Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes](https://mlsanity.com/paper/2102.09385/) (2021)
- [Sharp Analysis of Stochastic Optimization under Global Kurdyka-Lojasiewicz Inequality](https://mlsanity.com/paper/2210.01748/) (2022)
- [Convergence of the Forward-Backward Algorithm: Beyond the Worst Case with the Help of Geometry](https://mlsanity.com/paper/1703.09477/) (2017)
- [Explicit Regularization of Stochastic Gradient Methods through Duality](https://mlsanity.com/paper/2003.13807/) (2020)
- [Statistical Learning and Inverse Problems: A Stochastic Gradient Approach](https://mlsanity.com/paper/2209.14967/) (2022)
- [On the Convergence of Stochastic Gradient Descent for Nonlinear Ill-Posed Problems](https://mlsanity.com/paper/1907.03132/) (2019)
- [On the Saturation Phenomenon of Stochastic Gradient Descent for Linear Inverse Problems](https://mlsanity.com/paper/2010.10916/) (2020)
- [A Guide to Stochastic Optimisation for Large-Scale Inverse Problems](https://mlsanity.com/paper/2406.06342/) (2024)
- [Combining fast inertial dynamics for convex optimization with Tikhonov regularization](https://mlsanity.com/paper/1602.01973/) (2016)
- [Convex optimization via inertial algorithms with vanishing Tikhonov regularization: fast convergence to the minimum norm solution](https://mlsanity.com/paper/2104.11987/) (2021)
- [Splitting forward-backward penalty scheme for constrained variational problems](https://mlsanity.com/paper/1408.0974/) (2014)
- [Accelerated gradient methods with strong convergence to the minimum norm minimizer: a dynamic approach combining time scaling, averaging, and Tikhonov regularization](https://mlsanity.com/paper/2211.10140/) (2022)
- [Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces](https://mlsanity.com/paper/2403.06708/) (2024)

## Built on by (most cited first, 3 of 3)

- [The Role of Target Update Frequencies in Q-Learning](https://mlsanity.com/paper/2602.03911/) (2026)
- [Softmax gradient policy for variance minimization and risk-averse multi armed bandits](https://mlsanity.com/paper/2604.00241/) (2026)
- [Vanishing L2 regularization for the softmax Multi Armed Bandit](https://mlsanity.com/paper/2605.03752/) (2026)

## Abstract (arXiv)

The present article studies the minimization of convex, L-smooth functions defined on a separable real Hilbert space. We analyze regularized stochastic gradient descent (reg-SGD), a variant of stochastic gradient descent that uses a Tikhonov regularization with time-dependent, vanishing regularization parameter. We prove strong convergence of reg-SGD to the minimum-norm solution of the original problem without additional boundedness assumptions. Moreover, we quantify the rate of convergence and optimize the interplay between step-sizes and regularization decay. Our analysis reveals how vanishing Tikhonov regularization controls the flow of SGD and yields stable learning dynamics, offering new insights into the design of iterative algorithms for convex problems, including those that arise in ill-posed inverse problems. We validate our theoretical findings through numerical experiments on image reconstruction and ODE-based inverse problems.