# Flatness is a False Friend

arXiv: https://arxiv.org/abs/2006.09091 · HTML: https://mlsanity.com/paper/2006.09091/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 19 of 19)

- [Adaptive Gradient Methods at the Edge of Stability](https://mlsanity.com/paper/2207.14484/) (2022)
- [A Modern Look at the Relationship between Sharpness and Generalization](https://mlsanity.com/paper/2302.07011/) (2023)
- [On the Maximum Hessian Eigenvalue and Generalization](https://mlsanity.com/paper/2206.10654/) (2022)
- [Taxonomizing local versus global structure in neural network loss landscapes](https://mlsanity.com/paper/2107.11228/) (2021)
- [On the Implicit Bias of Adam](https://mlsanity.com/paper/2309.00079/) (2023)
- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [On progressive sharpening, flat minima and generalisation](https://mlsanity.com/paper/2305.14683/) (2023)
- [Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks](https://mlsanity.com/paper/2411.02139/) (2024)
- [A spin-glass model for the loss surfaces of generative adversarial networks](https://mlsanity.com/paper/2101.02524/) (2021)
- [Bringing the Discussion of Minima Sharpness to the Audio Domain: a Filter-Normalised Evaluation for Acoustic Scene Classification](https://mlsanity.com/paper/2309.16369/) (2023)
- [The instabilities of large learning rate training: a loss landscape view](https://mlsanity.com/paper/2307.11948/) (2023)
- [Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance](https://mlsanity.com/paper/2407.12996/) (2024)
- [In Search of the Successful Interpolation: On the Role of Sharpness in CLIP Generalization](https://mlsanity.com/paper/2410.16476/) (2024)
- [Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima](https://mlsanity.com/paper/2607.08380/) (2026)
- [Post-Training Science for Supervised Fine-Tuning](https://mlsanity.com/paper/2609.01244/) (2026)
- [SANE: The phases of gradient descent through Sharpness Adjusted Number of Effective parameters](https://mlsanity.com/paper/2305.18490/) (2023)
- [A simple connection from loss flatness to compressed neural representations](https://mlsanity.com/paper/2310.01770/) (2023)
- [Generalization of the Gibbs algorithm with high probability at low temperatures](https://mlsanity.com/paper/2502.11071/) (2025)
- [Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition](https://mlsanity.com/paper/2510.27651/) (2025)

## Abstract (arXiv)

Hessian based measures of flatness, such as the trace, Frobenius and spectral norms, have been argued, used and shown to relate to generalisation. In this paper we demonstrate that for feed forward neural networks under the cross entropy loss, we would expect low loss solutions with large weights to have small Hessian based measures of flatness. This implies that solutions obtained using L2 regularisation should in principle be sharper than those without, despite generalising better. We show this to be true for logistic regression, multi-layer perceptrons, simple convolutional, pre-activated and wide residual networks on the MNIST and CIFAR-100 datasets. Furthermore, we show that for adaptive optimisation algorithms using iterate averaging, on the VGG-16 network and CIFAR-100 dataset, achieve superior generalisation to SGD but are 30 × sharper. This theoretical finding, along with experimental results, raises serious questions about the validity of Hessian based sharpness measures in the discussion of generalisation. We further show that the Hessian rank can be bounded by the a constant times number of neurons multiplied by the number of classes, which in practice is often a small fraction of the network parameters. This explains the curious observation that many Hessian eigenvalues are either zero or very near zero which has been reported in the literature.