The Expected Loss of Preconditioned Langevin Dynamics Reveals the Hessian Rank
Not on the map. Why?
Abstract (arXiv)
Langevin dynamics (LD) is widely used for sampling from distributions and for optimization. In this work, we derive a closed-form expression for the expected loss of preconditioned LD near stationary points of the objective function. We use the fact that at the vicinity of such points, LD reduces to an Ornstein-Uhlenbeck process, which is amenable to convenient mathematical treatment. Our analysis reveals that when the preconditioning matrix satisfies a particular relation with respect to the noise covariance, LD's expected loss becomes proportional to the rank of the objective's Hessian. We illustrate the applicability of this result in the context of neural networks, where the Hessian rank has been shown to capture the complexity of the predictor function but is usually computationally hard to probe. Finally, we use our analysis to compare SGD-like and Adam-like preconditioners and identify the regimes under which each of them leads to a lower expected loss.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 21 Feb 2024.
Builds on
21 of 40 references.
References · 21
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,352
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
- 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
- 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
- 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
- 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
- 2015Neural GPUs Learn Algorithmscited by 118
- 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
- 2017Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimizationcited by 96
9 more
- 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
- 2015Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlocited by 61
- 2019Escaping Saddle Points with Adaptive Gradient Methodscited by 34
- 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
- 2020Analysis of Stochastic Gradient Descent in Continuous Timecited by 24
- 2019On Stationary-Point Hitting Time and Ergodicity of Stochastic Gradient Langevin Dynamicscited by 14
- 2016A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognitioncited by 10
- 2017Natural Langevin Dynamics for Neural Networkscited by 6
- 2020CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics with Simulated Annealingcited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.