# Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function arXiv: https://arxiv.org/abs/2107.08649 ยท HTML: https://mlsanity.com/paper/2107.08649/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 41 references that are in the map. - First posted on arXiv 19 Jul 2021. ## Builds on (most cited first, 21 of 21) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017) - [Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis](https://mlsanity.com/paper/1702.03849/) (2017) - [Theoretical guarantees for approximate sampling from smooth and log-concave densities](https://mlsanity.com/paper/1412.7392/) (2014) - [On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization](https://mlsanity.com/paper/1808.02941/) (2018) - [High-dimensional Bayesian inference via the Unadjusted Langevin Algorithm](https://mlsanity.com/paper/1605.01559/) (2016) - [Analysis of Langevin Monte Carlo via convex optimization](https://mlsanity.com/paper/1802.09188/) (2018) - [User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient](https://mlsanity.com/paper/1710.00095/) (2017) - [Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization](https://mlsanity.com/paper/1707.06618/) (2017) - [Efficient Bayesian computation by proximal Markov chain Monte Carlo: when Langevin meets Moreau](https://mlsanity.com/paper/1612.07471/) (2016) - [Sharp convergence rates for Langevin dynamics in the nonconvex setting](https://mlsanity.com/paper/1805.01648/) (2018) - [The promises and pitfalls of Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/1811.10072/) (2018) - [A statistical theory of cold posteriors in deep neural networks](https://mlsanity.com/paper/2008.05912/) (2020) - [Approximation Bounds for Random Neural Networks and Reservoir Systems](https://mlsanity.com/paper/2002.05933/) (2020) - [Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimization](https://mlsanity.com/paper/1910.02008/) (2019) - [The Tamed Unadjusted Langevin Algorithm](https://mlsanity.com/paper/1710.05559/) (2017) - [Deep neural networks, generic universal interpolation, and controlled ODEs](https://mlsanity.com/paper/1908.07838/) (2019) - [Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms](https://mlsanity.com/paper/2006.14514/) (2020) - [On stochastic gradient Langevin dynamics with dependent data streams in the logconcave case](https://mlsanity.com/paper/1812.02709/) (2018) - [On stochastic gradient Langevin dynamics with dependent data streams: the fully non-convex case](https://mlsanity.com/paper/1905.13142/) (2019) - [A fully data-driven approach to minimizing CVaR for portfolio of assets via SGLD with discontinuous updating](https://mlsanity.com/paper/2007.01672/) (2020) ## Built on by (most cited first, 10 of 10) - [Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data](https://mlsanity.com/paper/2402.14989/) (2024) - [Taming under isoperimetry](https://mlsanity.com/paper/2311.09003/) (2023) - [Langevin dynamics based algorithm e-THvarepsilonO POULA for stochastic optimization problems with discontinuous stochastic gradient](https://mlsanity.com/paper/2210.13193/) (2022) - [Unadjusted Langevin algorithm for sampling a mixture of weakly smooth potentials](https://mlsanity.com/paper/2112.09311/) (2021) - [Taming the Interacting Particle Langevin Algorithm: The Superlinear case](https://mlsanity.com/paper/2403.19587/) (2024) - [kTULA: A Langevin sampling algorithm with improved KL bounds under super-linear log-gradients](https://mlsanity.com/paper/2506.04878/) (2025) - [Global convergence of optimized adaptive importance samplers](https://mlsanity.com/paper/2201.00409/) (2022) - [Error estimates for tamed Euler and Randomized Euler schemes for SDEs with locally Lipschitz drift with applications to non-logconcave sampling and optimization](https://mlsanity.com/paper/2605.24937/) (2026) - [Deterministic Envelopes for Tamed SGLD: Decoupling Stochastic Gradient Noise and Localizing Taming](https://mlsanity.com/paper/2606.05242/) (2026) - [RELTA-SGLD: Relative-Growth Localized Taming for Nonconvex Stochastic-Gradient Langevin Learning](https://mlsanity.com/paper/2607.19544/) (2026) ## Abstract (arXiv) We consider non-convex stochastic optimization problems where the objective functions have super-linearly growing and discontinuous stochastic gradients. In such a setting, we provide a non-asymptotic analysis for the tamed unadjusted stochastic Langevin algorithm (TUSLA) introduced in Lovas et al. (2020). In particular, we establish non-asymptotic error bounds for the TUSLA algorithm in Wasserstein-1 and Wasserstein-2 distances. The latter result enables us to further derive non-asymptotic estimates for the expected excess risk. To illustrate the applicability of the main results, we consider an example from transfer learning with ReLU neural networks, which represents a key paradigm in machine learning. Numerical experiments are presented for the aforementioned example which support our theoretical findings. Hence, in this setting, we demonstrate both theoretically and numerically that the TUSLA algorithm can solve the optimization problem involving neural networks with ReLU activation function. Besides, we provide simulation results for synthetic examples where popular algorithms, e.g. ADAM, AMSGrad, RMSProp, and (vanilla) stochastic gradient descent (SGD) algorithm, may fail to find the minimizer of the objective functions due to the super-linear growth and the discontinuity of the corresponding stochastic gradient, while the TUSLA algorithm converges rapidly to the optimal solution. Moreover, we provide an empirical comparison of the performance of TUSLA with popular stochastic optimizers on real-world datasets, as well as investigate the effect of the key hyperparameters of TUSLA on its performance.