# Unveiling the structure of wide flat minima in neural networks

arXiv: https://arxiv.org/abs/2107.01163 · HTML: https://mlsanity.com/paper/2107.01163/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 37 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 7 of 7)

- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors](https://mlsanity.com/paper/1712.09376/) (2017)
- [Shaping the learning landscape in neural networks around wide flat minima](https://mlsanity.com/paper/1905.07833/) (2019)
- [Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes](https://mlsanity.com/paper/1605.06444/) (2016)
- [Local entropy as a measure for sampling solutions in Constraint Satisfaction Problems](https://mlsanity.com/paper/1511.05634/) (2015)
- [Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures](https://mlsanity.com/paper/2010.14761/) (2020)

## Built on by (most cited first, 28 of 28)

- [Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry](https://mlsanity.com/paper/2202.03038/) (2022)
- [Learning through atypical "phase transitions" in overparameterized neural networks](https://mlsanity.com/paper/2110.00683/) (2021)
- [Binary perceptron: efficient algorithms can find solutions in a rare well-connected cluster](https://mlsanity.com/paper/2111.03084/) (2021)
- [Typical and atypical solutions in non-convex neural networks with discrete and continuous weights](https://mlsanity.com/paper/2304.13871/) (2023)
- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [Do we really have to filter out random noise in pre-training data for language models?](https://mlsanity.com/paper/2502.06604/) (2025)
- [Phase transitions in the mini-batch size for sparse and dense two-layer neural networks](https://mlsanity.com/paper/2305.06435/) (2023)
- [Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks](https://mlsanity.com/paper/2410.06717/) (2024)
- [Complex Recurrent Spectral Network](https://mlsanity.com/paper/2312.07296/) (2023)
- [Performance of machine-learning-assisted Monte Carlo in sampling from simple statistical physics models](https://mlsanity.com/paper/2505.22598/) (2025)
- [Rare dense solutions clusters in asymmetric binary perceptrons -- local entropy via fully lifted RDT](https://mlsanity.com/paper/2506.19276/) (2025)
- [High-dimensional manifold of solutions in neural networks: insights from statistical physics](https://mlsanity.com/paper/2309.09240/) (2023)
- [Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting](https://mlsanity.com/paper/2308.01421/) (2023)
- [Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems](https://mlsanity.com/paper/2309.05337/) (2023)
- [Transfer Learning in Infinite Width Feature Learning Networks](https://mlsanity.com/paper/2507.04448/) (2025)
- [Binary perceptron computational gap -- a parametric fl RDT view](https://mlsanity.com/paper/2511.01037/) (2025)
- [Spring-block theory of feature learning in deep neural networks](https://mlsanity.com/paper/2407.19353/) (2024)
- [Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks](https://mlsanity.com/paper/2604.09412/) (2026)
- [Flat Minima in Linear Estimation and an Extended Gauss Markov Theorem](https://mlsanity.com/paper/2311.11093/) (2023)
- [The Curvature Rate λ: A Scalar Measure of Input-Space Sharpness in Neural Networks](https://mlsanity.com/paper/2511.01438/) (2025)
- [Parametric RDT approach to computational gap of symmetric binary perceptron](https://mlsanity.com/paper/2601.10628/) (2026)
- [Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networks](https://mlsanity.com/paper/2608.22361/) (2026)
- [Physics Inspired Approaches To Understanding Gaussian Processes](https://mlsanity.com/paper/2305.10748/) (2023)
- [Engineered Ordinary Differential Equations as Classification Algorithm (EODECA): thorough characterization and testing](https://mlsanity.com/paper/2312.14681/) (2023)
- [A large deviation view of stationarized fully lifted blirp interpolation](https://mlsanity.com/paper/2506.19273/) (2025)
- [Fractal basin geometry and the limits to predictability and reproducibility in deep learning](https://mlsanity.com/paper/2510.05606/) (2025)
- [A Function-Space Approach to the Statistical Mechanics of Learning Dynamics](https://mlsanity.com/paper/2609.09589/) (2026)
- [Symmetry-quotient Flatness and Generalization](https://mlsanity.com/paper/2609.31634/) (2026)

## Abstract (arXiv)

The success of deep learning has revealed the application potential of neural networks across the sciences and opened up fundamental theoretical problems. In particular, the fact that learning algorithms based on simple variants of gradient methods are able to find near-optimal minima of highly nonconvex loss functions is an unexpected feature of neural networks. Moreover, such algorithms are able to fit the data even in the presence of noise, and yet they have excellent predictive capabilities. Several empirical results have shown a reproducible correlation between the so-called flatness of the minima achieved by the algorithms and the generalization performance. At the same time, statistical physics results have shown that in nonconvex networks a multitude of narrow minima may coexist with a much smaller number of wide flat minima, which generalize well. Here we show that wide flat minima arise as complex extensive structures, from the coalescence of minima around "high-margin" (i.e., locally robust) configurations. Despite being exponentially rare compared to zero-margin ones, high-margin minima tend to concentrate in particular regions. These minima are in turn surrounded by other solutions of smaller and smaller margin, leading to dense regions of solutions over long distances. Our analysis also provides an alternative analytical method for estimating when flat minima appear and when algorithms begin to find solutions, as the number of model parameters varies.