mlsanityData to 7 Oct 2026

SANE: The phases of gradient descent through Sharpness Adjusted Number of Effective parameters

arXiv 2305.18490 ↗First posted 29 May 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern neural networks are undeniably successful. Numerous studies have investigated how the curvature of loss landscapes can affect the quality of solutions. In this work we consider the Hessian matrix during network training. We reiterate the connection between the number of "well-determined" or "effective" parameters and the generalisation performance of neural nets, and we demonstrate its use as a tool for model comparison. By considering the local curvature, we propose Sharpness Adjusted Number of Effective parameters (SANE), a measure of effective dimensionality for the quality of solutions. We show that SANE is robust to large learning rates, which represent learning regimes that are attractive but (in)famously unstable. We provide evidence and characterise the Hessian shifts across "loss basins" at large learning rates. Finally, extending our analysis to deeper neural networks, we provide an approximation to the full-network Hessian, exploiting the natural ordering of neural weights, and use this approximation to provide extensive empirical evidence for our claims.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

27 of 44 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  9. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  10. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  11. 2020Time Series Forecasting With Deep Learning: A Surveycited by 457
  12. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
15 more
  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  2. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  3. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  5. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  6. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  7. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  8. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  9. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  10. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  11. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 50
  12. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  13. 2017Early Stopping without a Validation Setcited by 39
  14. 2020Flatness is a False Friendcited by 19
  15. 2022Visualizing high-dimensional loss landscapes with Hessian directionscited by 16

Built on by

Not cited yet.