mlsanityData to 14 Oct 2026

Power-law Dynamic arising from machine learning

arXiv 2306.09624 ↗First posted 16 Jun 2023stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We study a kind of new SDE that was arisen from the research on optimization in machine learning, we call it power-law dynamic because its stationary distribution cannot have sub-Gaussian tail and obeys power-law. We prove that the power-law dynamic is ergodic with unique stationary distribution, provided the learning rate is small enough. We investigate its first exist time. In particular, we compare the exit times of the (continuous) power-law dynamic and its discretization. The comparison can help guide machine learning algorithm.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 35 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

17 of 35 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  6. 2017Three Factors Influencing Minima in SGDcited by 273
  7. 2016Dual Learning for Machine Translationcited by 148
  8. 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
  9. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  10. 2020The Heavy-Tail Phenomenon in SGDcited by 124
  11. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  12. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
5 more
  1. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  2. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 44
  3. 2020Dynamic of Stochastic Gradient Descent with State-Dependent Noisecited by 12
  4. 2020A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minimacited by 11
  5. 2018Differential Equations for Modeling Asynchronous Algorithmscited by 4

Built on by

Not cited yet.