mlsanityData to 7 Oct 2026

Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapes

arXiv 2409.11995 ↗First posted 18 Sep 2024cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

The loss landscape of neural networks is a critical aspect of their training, and understanding its properties is essential for improving their performance. In this paper, we investigate how the loss surface changes when the sample size increases, a previously unexplored issue. We theoretically analyze the convergence of the loss landscape in a fully connected neural network and derive upper bounds for the difference in loss function values when adding a new object to the sample. Our empirical study confirms these results on various datasets, demonstrating the convergence of the loss function surface for image classification tasks. Our findings provide insights into the local geometry of neural loss landscapes and have implications for the development of sample size determination techniques.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

37 of 51 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2017Exploring Generalization in Deep Learningcited by 595
  9. 2017Large Batch Training of Convolutional Networkscited by 529
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  12. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
25 more
  1. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  2. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  3. 2014The Loss Surfaces of Multilayer Networkscited by 278
  4. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  5. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  6. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  7. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  8. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  9. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  10. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  11. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  12. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  13. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  14. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  15. 2020A Comparison of Optimization Algorithms for Deep Learningcited by 23
  16. 2019Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretationcited by 22
  17. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 14
  18. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  19. 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 11
  20. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  21. 2023The Hessian perspective into the Nature of Convolutional Neural Networkscited by 10
  22. 2024Landscaping Linear Mode Connectivitycited by 7
  23. 2023On progressive sharpening, flat minima and generalisationcited by 6
  24. 2023The instabilities of large learning rate training: a loss landscape viewcited by 3
  25. 2024Agnostic Sharpness-Aware Minimizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Curvature-Aligned Probing for Local Loss-Landscape Stabilizationcited by 0