mlsanityData to 14 Oct 2026

Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks

arXiv 2105.13937 ↗First posted 28 May 2021cs.LG math.OC math.PRcited by 11

Not on the map. Why?

Abstract (arXiv)

We present a new class of Langevin based algorithms, which overcomes many of the known shortcomings of popular adaptive optimizers that are currently used for the fine tuning of deep learning models. Its underpinning theory relies on recent advances of Euler's polygonal approximations for stochastic differential equations (SDEs) with monotone coefficients. As a result, it inherits the stability properties of tamed algorithms, while it addresses other known issues, e.g. vanishing gradients in neural networks. In particular, we provide a nonasymptotic analysis and full theoretical guarantees for the convergence properties of an algorithm of this novel class, which we named THvarepsilonO POULA (or, simply, TheoPouLa). Finally, several experiments are presented with different types of deep learning models, which show the superior performance of TheoPouLa over many popular adaptive optimization algorithms.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 31 of its 60 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

31 of 60 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2019On the Convergence of Adam and Beyondcited by 841
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  8. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  9. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  10. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 243
  11. 2014Theoretical guarantees for approximate sampling from smooth and log-concave densitiescited by 208
  12. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
19 more
  1. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  2. 2015Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithmcited by 197
  3. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  4. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  5. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  6. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  7. 2017Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimizationcited by 97
  8. 2018The promises and pitfalls of Stochastic Gradient Langevin Dynamicscited by 56
  9. 2020A statistical theory of cold posteriors in deep neural networkscited by 54
  10. 2018Global Non-convex Optimization with Discretized Diffusionscited by 44
  11. 2019Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimizationcited by 38
  12. 2020Non-convex Learning via Replica Exchange Stochastic Gradient MCMCcited by 34
  13. 2017Beyond Log-concavity: Provable Guarantees for Sampling Multi-modal Distributions using Simulated Tempering Langevin Monte Carlocited by 33
  14. 2020Faster Convergence of Stochastic Gradient Langevin Dynamics for Non-Log-Concave Samplingcited by 32
  15. 2017The Tamed Unadjusted Langevin Algorithmcited by 31
  16. 2020A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributionscited by 26
  17. 2020Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithmscited by 21
  18. 2018On stochastic gradient Langevin dynamics with dependent data streams in the logconcave casecited by 20
  19. 2019On stochastic gradient Langevin dynamics with dependent data streams: the fully non-convex casecited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 2; 2025: 3; 2026: 2.

11 papers.

  1. 2024Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Datacited by 36
  2. 2023On diffusion-based generative models and their error bounds: The log-concave case with full convergence estimatescited by 25
  3. 2025Wasserstein Convergence of Score-based Generative Models under Semiconvexity and Discontinuous Gradientscited by 15
  4. 2023Taming under isoperimetrycited by 7
  5. 2022Langevin dynamics based algorithm e-THvarepsilonO POULA for stochastic optimization problems with discontinuous stochastic gradientcited by 5
  6. 2025The Performance Of The Unadjusted Langevin Algorithm Without Smoothness Assumptionscited by 4
  7. 2024Taming the Interacting Particle Langevin Algorithm: The Superlinear casecited by 3
  8. 2022Global convergence of optimized adaptive importance samplerscited by 2
  9. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0
  10. 2026RELTA-SGLD: Relative-Growth Localized Taming for Nonconvex Stochastic-Gradient Langevin Learningcited by 0
  11. 2026Muon meets Tamed Langevin: Momentum Preconditioning beyond Convex and gradient-Lipschitz Potentialscited by 0