# Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms

arXiv: https://arxiv.org/abs/2006.14514 · HTML: https://mlsanity.com/paper/2006.14514/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2020.

## Built on by (most cited first, 21 of 21)

- [A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions](https://mlsanity.com/paper/2102.09924/) (2021)
- [Kinetic Langevin MCMC Sampling Without Gradient Lipschitz Continuity -- the Strongly Convex Case](https://mlsanity.com/paper/2301.08039/) (2023)
- [Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks](https://mlsanity.com/paper/2105.13937/) (2021)
- [Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function](https://mlsanity.com/paper/2107.08649/) (2021)
- [A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions](https://mlsanity.com/paper/2108.04620/) (2021)
- [A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2104.00277/) (2021)
- [Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2112.07369/) (2021)
- [Taming under isoperimetry](https://mlsanity.com/paper/2311.09003/) (2023)
- [Tamed Langevin sampling under weaker conditions](https://mlsanity.com/paper/2405.17693/) (2024)
- [Langevin dynamics based algorithm e-THvarepsilonO POULA for stochastic optimization problems with discontinuous stochastic gradient](https://mlsanity.com/paper/2210.13193/) (2022)
- [Statistical Finite Elements via Langevin Dynamics](https://mlsanity.com/paper/2110.11131/) (2021)
- [The Performance Of The Unadjusted Langevin Algorithm Without Smoothness Assumptions](https://mlsanity.com/paper/2502.03458/) (2025)
- [Unadjusted Langevin algorithm for sampling a mixture of weakly smooth potentials](https://mlsanity.com/paper/2112.09311/) (2021)
- [Taming the Interacting Particle Langevin Algorithm: The Superlinear case](https://mlsanity.com/paper/2403.19587/) (2024)
- [kTULA: A Langevin sampling algorithm with improved KL bounds under super-linear log-gradients](https://mlsanity.com/paper/2506.04878/) (2025)
- [Error estimates for tamed Euler and Randomized Euler schemes for SDEs with locally Lipschitz drift with applications to non-logconcave sampling and optimization](https://mlsanity.com/paper/2605.24937/) (2026)
- [Deterministic Envelopes for Tamed SGLD: Decoupling Stochastic Gradient Noise and Localizing Taming](https://mlsanity.com/paper/2606.05242/) (2026)
- [Contractive kinetic Langevin samplers beyond global Lipschitz continuity](https://mlsanity.com/paper/2509.12031/) (2025)
- [Deterministic Denominator Design for Localized Tamed Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2606.10559/) (2026)
- [Flatness-Aware Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2510.02174/) (2025)
- [RELTA-SGLD: Relative-Growth Localized Taming for Nonconvex Stochastic-Gradient Langevin Learning](https://mlsanity.com/paper/2607.19544/) (2026)

## Abstract (arXiv)

Artificial neural networks (ANNs) are typically highly nonlinear systems which are finely tuned via the optimization of their associated, non-convex loss functions. In many cases, the gradient of any such loss function has superlinear growth, making the use of the widely-accepted (stochastic) gradient descent methods, which are based on Euler numerical schemes, problematic. We offer a new learning algorithm based on an appropriately constructed variant of the popular stochastic gradient Langevin dynamics (SGLD), which is called tamed unadjusted stochastic Langevin algorithm (TUSLA). We also provide a nonasymptotic analysis of the new algorithm's convergence properties in the context of non-convex learning problems with the use of ANNs. Thus, we provide finite-time guarantees for TUSLA to find approximate minimizers of both empirical and population risks. The roots of the TUSLA algorithm are based on the taming technology for diffusion processes with superlinear coefficients as developed in citettamed-euler, SabanisAoAP and for MCMC algorithms in citettula. Numerical experiments are presented which confirm the theoretical findings and illustrate the need for the use of the new algorithm in comparison to vanilla SGLD within the framework of ANNs.