mlsanityData to 7 Oct 2026

The asymptotic spectrum of the Hessian of DNN throughout training

arXiv 1910.02875 ↗First posted 1 Oct 2019cs.LG cs.NE stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. In the so-called mean-field limit, where the NTK is not fixed during training, we describe the first two moments of the Hessian at initialization.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 4; 2023: 1; 2024: 3; 2025: 3; 2026: 2.

19 papers.

Sort:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  2. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  3. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  4. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  5. 2021Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learningcited by 23
  6. 2022Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learningcited by 22
  7. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  8. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  9. 2021Learning with Gradient Descent and Weakly Convex Lossescited by 16
  10. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 14
  11. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  12. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
7 more
  1. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  2. 2024Theoretical characterisation of the Gauss-Newton conditioning in Neural Networkscited by 5
  3. 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
  4. 2025Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvgcited by 2
  5. 2026On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvaturecited by 2
  6. 2025Mathematical Foundations of Neural Tangents and Infinite-Width Networkscited by 1
  7. 2026Boosting Data Augmentation with Stochastic Weight Averagingcited by 0