# A Scale Invariant Flatness Measure for Deep Network Minima

arXiv: https://arxiv.org/abs/1902.02434 · HTML: https://mlsanity.com/paper/1902.02434/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2019.

## Built on by (most cited first, 8 of 8)

- [On PAC-Bayesian reconstruction guarantees for VAEs](https://mlsanity.com/paper/2202.11455/) (2022)
- [Why flatness does and does not correlate with generalization for deep neural networks](https://mlsanity.com/paper/2103.06219/) (2021)
- [Exponential escape efficiency of SGD from sharp minima in non-stationary regime](https://mlsanity.com/paper/2111.04004/) (2021)
- [How You Start Matters for Generalization](https://mlsanity.com/paper/2206.08558/) (2022)
- [Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It](https://mlsanity.com/paper/2505.05409/) (2025)
- [Minimum sharpness: Scale-invariant parameter-robustness of neural networks](https://mlsanity.com/paper/2106.12612/) (2021)
- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)
- [A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks](https://mlsanity.com/paper/2606.18303/) (2026)

## Abstract (arXiv)

It has been empirically observed that the flatness of minima obtained from training deep networks seems to correlate with better generalization. However, for deep networks with positively homogeneous activations, most measures of sharpness/flatness are not invariant to rescaling of the network parameters, corresponding to the same function. This means that the measure of flatness/sharpness can be made as small or as large as possible through rescaling, rendering the quantitative measures meaningless. In this paper we show that for deep networks with positively homogenous activations, these rescalings constitute equivalence relations, and that these equivalence relations induce a quotient manifold structure in the parameter space. Using this manifold structure and an appropriate metric, we propose a Hessian-based measure for flatness that is invariant to rescaling. We use this new measure to confirm the proposition that Large-Batch SGD minima are indeed sharper than Small-Batch SGD minima.