# On the Maximum Hessian Eigenvalue and Generalization

arXiv: https://arxiv.org/abs/2206.10654 · HTML: https://mlsanity.com/paper/2206.10654/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 21 Jun 2022.

## Builds on (most cited first, 24 of 24)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Three Factors Influencing Minima in SGD](https://mlsanity.com/paper/1711.04623/) (2017)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [The large learning rate phase of deep learning: the catapult mechanism](https://mlsanity.com/paper/2003.02218/) (2020)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [The Break-Even Point on Optimization Trajectories of Deep Neural Networks](https://mlsanity.com/paper/2002.09572/) (2020)
- [Towards Understanding Sharpness-Aware Minimization](https://mlsanity.com/paper/2206.06232/) (2022)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [In Search of Robust Measures of Generalization](https://mlsanity.com/paper/2010.11924/) (2020)
- [Flatness is a False Friend](https://mlsanity.com/paper/2006.09091/) (2020)
- [An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise](https://mlsanity.com/paper/1902.08234/) (2019)
- [Why flatness does and does not correlate with generalization for deep neural networks](https://mlsanity.com/paper/2103.06219/) (2021)
- [Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problems](https://mlsanity.com/paper/2005.07360/) (2020)

## Built on by (most cited first, 30 of 57)

- [A Modern Look at the Relationship between Sharpness and Generalization](https://mlsanity.com/paper/2302.07011/) (2023)
- [Why are Sensitive Functions Hard for Transformers?](https://mlsanity.com/paper/2402.09963/) (2024)
- [SGD with Large Step Sizes Learns Sparse Features](https://mlsanity.com/paper/2210.05337/) (2022)
- [Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization](https://mlsanity.com/paper/2303.03108/) (2023)
- [PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning](https://mlsanity.com/paper/2306.10711/) (2023)
- [Early Weight Averaging meets High Learning Rates for LLM Pre-training](https://mlsanity.com/paper/2306.03241/) (2023)
- [Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home](https://mlsanity.com/paper/2501.12835/) (2025)
- [Flatness-Aware Minimization for Domain Generalization](https://mlsanity.com/paper/2307.11108/) (2023)
- [The Crucial Role of Normalization in Sharpness-Aware Minimization](https://mlsanity.com/paper/2305.15287/) (2023)
- [How to escape sharp minima with random perturbations](https://mlsanity.com/paper/2305.15659/) (2023)
- [Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning](https://mlsanity.com/paper/2603.16127/) (2026)
- [The Interpolating Information Criterion for Overparameterized Models](https://mlsanity.com/paper/2307.07785/) (2023)
- [On a continuous time model of gradient descent dynamics and instability in deep learning](https://mlsanity.com/paper/2302.01952/) (2023)
- [Forget Forgetting: Continual Learning in a World of Abundant Memory](https://mlsanity.com/paper/2502.07274/) (2025)
- [mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization](https://mlsanity.com/paper/2302.09693/) (2023)
- [CR-SAM: Curvature Regularized Sharpness-Aware Minimization](https://mlsanity.com/paper/2312.13555/) (2023)
- [Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes](https://mlsanity.com/paper/2412.13573/) (2024)
- [Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning](https://mlsanity.com/paper/2405.20439/) (2024)
- [Catapult Dynamics and Phase Transitions in Quadratic Nets](https://mlsanity.com/paper/2301.07737/) (2023)
- [A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs](https://mlsanity.com/paper/2601.16979/) (2026)
- [Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equations](https://mlsanity.com/paper/2411.15111/) (2024)
- [Where Do Large Learning Rates Lead Us?](https://mlsanity.com/paper/2410.22113/) (2024)
- [Transformer-Based Approaches for Sensor-Based Human Activity Recognition: Opportunities and Challenges](https://mlsanity.com/paper/2410.13605/) (2024)
- [A Basin-Selection Perspective on Grokking via Singular Learning Theory](https://mlsanity.com/paper/2603.01192/) (2026)
- [The instabilities of large learning rate training: a loss landscape view](https://mlsanity.com/paper/2307.11948/) (2023)
- [Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization](https://mlsanity.com/paper/2404.17768/) (2024)
- [Tractable Sharpness-Aware Learning of Probabilistic Circuits](https://mlsanity.com/paper/2508.05537/) (2025)
- [When Flatness Does (Not) Guarantee Adversarial Robustness](https://mlsanity.com/paper/2510.14231/) (2025)
- [Towards a Better Understanding of Learning with Multiagent Teams](https://mlsanity.com/paper/2306.16205/) (2023)
- [On The Fairness Impacts of Hardware Selection in Machine Learning](https://mlsanity.com/paper/2312.03886/) (2023)

## Abstract (arXiv)

The mechanisms by which certain training interventions, such as increasing learning rates and applying batch normalization, improve the generalization of deep networks remains a mystery. Prior works have speculated that "flatter" solutions generalize better than "sharper" solutions to unseen data, motivating several metrics for measuring flatness (particularly λmax, the largest eigenvalue of the Hessian of the loss); and algorithms, such as Sharpness-Aware Minimization (SAM) [1], that directly optimize for flatness. Other works question the link between λmax and generalization. In this paper, we present findings that call λmax's influence on generalization further into question. We show that: (1) while larger learning rates reduce λmax for all batch sizes, generalization benefits sometimes vanish at larger batch sizes; (2) by scaling batch size and learning rate simultaneously, we can change λmax without affecting generalization; (3) while SAM produces smaller λmax for all batch sizes, generalization benefits (also) vanish with larger batch sizes; (4) for dropout, excessively high dropout probabilities can degrade generalization, even as they promote smaller λmax; and (5) while batch-normalization does not consistently produce smaller λmax, it nevertheless confers generalization benefits. While our experiments affirm the generalization benefits of large learning rates and SAM for minibatch SGD, the GD-SGD discrepancy demonstrates limits to λmax's ability to explain generalization in neural networks.