# Relative Flatness and Generalization

arXiv: https://arxiv.org/abs/2001.00939 · HTML: https://mlsanity.com/paper/2001.00939/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jan 2020.

## Built on by (most cited first, 30 of 88)

- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [A Modern Look at the Relationship between Sharpness and Generalization](https://mlsanity.com/paper/2302.07011/) (2023)
- [When Do Flat Minima Optimizers Work?](https://mlsanity.com/paper/2202.00661/) (2022)
- [Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization](https://mlsanity.com/paper/2303.03108/) (2023)
- [The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent](https://mlsanity.com/paper/2305.17490/) (2023)
- [Out-of-distribution Detection with Implicit Outlier Transformation](https://mlsanity.com/paper/2303.05033/) (2023)
- [An Adaptive Policy to Employ Sharpness-Aware Minimization](https://mlsanity.com/paper/2304.14647/) (2023)
- [Anticorrelated Noise Injection for Improved Generalization](https://mlsanity.com/paper/2202.02831/) (2022)
- [Symmetries, flat minima, and the conserved quantities of gradient flow](https://mlsanity.com/paper/2210.17216/) (2022)
- [Normalization Layers Are All That Sharpness-Aware Minimization Needs](https://mlsanity.com/paper/2306.04226/) (2023)
- [Flatness-Aware Minimization for Domain Generalization](https://mlsanity.com/paper/2307.11108/) (2023)
- [Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Power](https://mlsanity.com/paper/2205.13863/) (2022)
- [Improving Convergence and Generalization Using Parameter Symmetries](https://mlsanity.com/paper/2305.13404/) (2023)
- [Layer-wise Linear Mode Connectivity](https://mlsanity.com/paper/2307.06966/) (2023)
- [DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networks](https://mlsanity.com/paper/2302.14685/) (2023)
- [Bounding the Rademacher Complexity of Fourier neural operators](https://mlsanity.com/paper/2209.05150/) (2022)
- [The Geometry of Neural Nets' Parameter Spaces Under Reparametrization](https://mlsanity.com/paper/2302.07384/) (2023)
- [Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspective](https://mlsanity.com/paper/2211.13723/) (2022)
- [Towards Lightweight Black-Box Attacks against Deep Neural Networks](https://mlsanity.com/paper/2209.14826/) (2022)
- [A PAC-Bayesian Link Between Generalisation and Flat Minima](https://mlsanity.com/paper/2402.08508/) (2024)
- [Towards Robust Out-of-Distribution Generalization Bounds via Sharpness](https://mlsanity.com/paper/2403.06392/) (2024)
- [Towards fully covariant machine learning](https://mlsanity.com/paper/2301.13724/) (2023)
- [Flat Seeking Bayesian Neural Networks](https://mlsanity.com/paper/2302.02713/) (2023)
- [On the robustness of randomized classifiers to adversarial examples](https://mlsanity.com/paper/2102.10875/) (2021)
- [How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective](https://mlsanity.com/paper/2306.05857/) (2023)
- [Frequency-Masked Embedding Inference: A Non-Contrastive Approach for Time Series Representation Learning](https://mlsanity.com/paper/2412.20790/) (2024)
- [Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking](https://mlsanity.com/paper/2509.17738/) (2025)
- [Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernel](https://mlsanity.com/paper/2209.15208/) (2022)
- [On progressive sharpening, flat minima and generalisation](https://mlsanity.com/paper/2305.14683/) (2023)
- [Singular-limit analysis of gradient descent with noise injection](https://mlsanity.com/paper/2404.12293/) (2024)

## Abstract (arXiv)

Flatness of the loss curve is conjectured to be connected to the generalization ability of machine learning models, in particular neural networks. While it has been empirically observed that flatness measures consistently correlate strongly with generalization, it is still an open theoretical problem why and under which circumstances flatness is connected to generalization, in particular in light of reparameterizations that change certain flatness measures but leave generalization unchanged. We investigate the connection between flatness and generalization by relating it to the interpolation from representative data, deriving notions of representativeness, and feature robustness. The notions allow us to rigorously connect flatness and generalization and to identify conditions under which the connection holds. Moreover, they give rise to a novel, but natural relative flatness measure that correlates strongly with generalization, simplifies to ridge regression for ordinary least squares, and solves the reparameterization issue.