mlsanityData to 7 Oct 2026

The Role of the Time-Dependent Hessian in High-Dimensional Optimization

arXiv 2403.02418 ↗First posted 4 Mar 2024cs.LG cond-mat.dis-nn cond-mat.stat-mechcited by 7

Not on the map. Why?

Abstract (arXiv)

Gradient descent is commonly used to find minima in rough landscapes, particularly in recent machine learning applications. However, a theoretical understanding of why good solutions are found remains elusive, especially in strongly non-convex and high-dimensional settings. Here, we focus on the phase retrieval problem as a typical example, which has received a lot of attention recently in theoretical machine learning. We analyze the Hessian during gradient descent, identify a dynamical transition in its spectral properties, and relate it to the ability of escaping rough regions in the loss landscape. When the signal-to-noise ratio (SNR) is large enough, an informative negative direction exists in the Hessian at the beginning of the descent, i.e in the initial condition. While descending, a BBP transition in the spectrum takes place in finite time: the direction is lost, and the dynamics is trapped in a rugged region filled with marginally stable bad minima. Surprisingly, for finite system sizes, this window of negative curvature allows the system to recover the signal well before the theoretical SNR found for infinite sizes, emphasizing the central role of initialization and early-time dynamics for efficiently navigating rough landscapes.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 20 of its 49 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

20 of 49 references.

References · 20

  1. 2017Exploring Generalization in Deep Learningcited by 595
  2. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  3. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  4. 2018To understand deep learning we need to understand kernel learningcited by 186
  5. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  6. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  7. 2016A Geometric Analysis of Phase Retrievalcited by 92
  8. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  9. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  10. 2015Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systemscited by 50
  11. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  12. 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42
8 more
  1. 2017Fundamental Limits of Weak Recovery with Applications to Phase Retrievalcited by 39
  2. 2019Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Modelscited by 28
  3. 2017Phase Transitions of Spectral Initialization for High-Dimensional Nonconvex Estimationcited by 26
  4. 2021Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemcited by 23
  5. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  6. 2023On Single Index Models beyond Gaussian Datacited by 17
  7. 2019Landscape Complexity for the Empirical Risk of Generalized Linear Modelscited by 15
  8. 2021Solving phase retrieval with random initial guess is nearly as good as by spectral initializationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Optimal scaling laws in learning hierarchical multi-index modelscited by 6
  2. 2026Deep Learning of Compositional Targets with Hierarchical Spectral Methodscited by 2
  3. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  4. 2026Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenariocited by 2
  5. 2026Spectral phase transitions and trainability in neural network learning dynamicscited by 1
  6. 2026Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrievalcited by 0
  7. 2026Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localizationcited by 0