# RELTA-SGLD: Relative-Growth Localized Taming for Nonconvex Stochastic-Gradient Langevin Learning

arXiv: https://arxiv.org/abs/2607.19544 · HTML: https://mlsanity.com/paper/2607.19544/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 21 Jul 2026.

## Builds on (most cited first, 14 of 14)

- [Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis](https://mlsanity.com/paper/1702.03849/) (2017)
- [Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization](https://mlsanity.com/paper/1707.06618/) (2017)
- [The Tamed Unadjusted Langevin Algorithm](https://mlsanity.com/paper/1710.05559/) (2017)
- [Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms](https://mlsanity.com/paper/2006.14514/) (2020)
- [Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks](https://mlsanity.com/paper/2105.13937/) (2021)
- [Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function](https://mlsanity.com/paper/2107.08649/) (2021)
- [Non-asymptotic convergence bounds for modified tamed unadjusted Langevin algorithm in non-convex setting](https://mlsanity.com/paper/2207.02600/) (2022)
- [Taming under isoperimetry](https://mlsanity.com/paper/2311.09003/) (2023)
- [Tamed Langevin sampling under weaker conditions](https://mlsanity.com/paper/2405.17693/) (2024)
- [Langevin dynamics based algorithm e-THvarepsilonO POULA for stochastic optimization problems with discontinuous stochastic gradient](https://mlsanity.com/paper/2210.13193/) (2022)
- [kTULA: A Langevin sampling algorithm with improved KL bounds under super-linear log-gradients](https://mlsanity.com/paper/2506.04878/) (2025)
- [Error estimates for tamed Euler and Randomized Euler schemes for SDEs with locally Lipschitz drift with applications to non-logconcave sampling and optimization](https://mlsanity.com/paper/2605.24937/) (2026)
- [Deterministic Envelopes for Tamed SGLD: Decoupling Stochastic Gradient Noise and Localizing Taming](https://mlsanity.com/paper/2606.05242/) (2026)
- [Deterministic Denominator Design for Localized Tamed Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2606.10559/) (2026)

## Abstract (arXiv)

We introduce RELTA-SGLD, a taming scheme that stabilizes superlinear stochastic-gradient updates while reducing unnecessary suppression of the original learning drift. A threshold determines where the taming turns on, while a relative-growth principle derived from the one-step Lyapunov stability condition determines the required taming strength. Together, they produce a lighter λ-scale denominator and preserve a nonvanishing far-tail return. As a consequence, we prove polynomial moment stability and first-order stationary accuracy in both W1 and W2 for nonconvex SGLD with superlinearly growing stochastic-gradient oracles, improving the corresponding half-order and quarter-order bounds for comparable stochastic-gradient tamed schemes. On Fashion-MNIST under active stabilization pressure, RELTA improves the mean learning metrics over both untamed SGLD and TUSLA and remains competitive with a tuned AdamW reference. In an ordinary-training regime, its lighter localized denominator reduces unnecessary perturbation of the original update and maintains nearly untamed learning dynamics.