# Entropic gradient descent algorithms and wide flat minima

arXiv: https://arxiv.org/abs/2006.07897 · HTML: https://mlsanity.com/paper/2006.07897/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2020.

## Built on by (most cited first, 25 of 25)

- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima](https://mlsanity.com/paper/2111.01549/) (2021)
- [Rethinking Confidence Calibration for Failure Prediction](https://mlsanity.com/paper/2303.02970/) (2023)
- [Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry](https://mlsanity.com/paper/2202.03038/) (2022)
- [Revisiting Self-Distillation](https://mlsanity.com/paper/2206.08491/) (2022)
- [Learning through atypical "phase transitions" in overparameterized neural networks](https://mlsanity.com/paper/2110.00683/) (2021)
- [Visualizing high-dimensional loss landscapes with Hessian directions](https://mlsanity.com/paper/2208.13219/) (2022)
- [Deep learning via message passing algorithms based on belief propagation](https://mlsanity.com/paper/2110.14583/) (2021)
- [Revisiting Confidence Estimation: Towards Reliable Failure Prediction](https://mlsanity.com/paper/2403.02886/) (2024)
- [An Expectation-Maximization Perspective on Federated Learning](https://mlsanity.com/paper/2111.10192/) (2021)
- [Phase transitions in the mini-batch size for sparse and dense two-layer neural networks](https://mlsanity.com/paper/2305.06435/) (2023)
- [Understanding the effect of sparsity on neural networks robustness](https://mlsanity.com/paper/2206.10915/) (2022)
- [Bias-variance decomposition of overparameterized regression with random linear features](https://mlsanity.com/paper/2203.05443/) (2022)
- [Entropy-MCMC: Sampling from Flat Basins with Ease](https://mlsanity.com/paper/2310.05401/) (2023)
- [FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness](https://mlsanity.com/paper/2402.19102/) (2024)
- [On the curvature of the loss landscape](https://mlsanity.com/paper/2307.04719/) (2023)
- [On Cyclical MCMC Sampling](https://mlsanity.com/paper/2403.00230/) (2024)
- [Architecture-Aware Minimization (A2M): How to Find Flat Minima in Neural Architecture Search](https://mlsanity.com/paper/2503.10404/) (2025)
- [Non-Convex Optimization with Spectral Radius Regularization](https://mlsanity.com/paper/2102.11210/) (2021)
- [SEAL: Searching Expandable Architectures for Incremental Learning](https://mlsanity.com/paper/2505.10457/) (2025)
- [Visualizing the loss landscapes of physics-informed neural networks](https://mlsanity.com/paper/2602.05849/) (2026)
- [Controlled Langevin Dynamics for Sampling of Feedforward Neural Networks Trained with Minibatches](https://mlsanity.com/paper/2603.15367/) (2026)
- [Low-power multi-mode fiber projector overcomes shallow neural networks classifiers](https://mlsanity.com/paper/2210.04745/) (2022)
- [Motivation is Something You Need](https://mlsanity.com/paper/2602.21064/) (2026)
- [Improving Generalization by Permutation Routing Across Model Copies](https://mlsanity.com/paper/2605.09256/) (2026)

## Abstract (arXiv)

The properties of flat minima in the empirical risk landscape of neural networks have been debated for some time. Increasing evidence suggests they possess better generalization capabilities with respect to sharp ones. First, we discuss Gaussian mixture classification models and show analytically that there exist Bayes optimal pointwise estimators which correspond to minimizers belonging to wide flat regions. These estimators can be found by applying maximum flatness algorithms either directly on the classifier (which is norm independent) or on the differentiable loss function used in learning. Next, we extend the analysis to the deep learning scenario by extensive numerical validations. Using two algorithms, Entropy-SGD and Replicated-SGD, that explicitly include in the optimization objective a non-local flatness measure known as local entropy, we consistently improve the generalization error for common architectures (e.g. ResNet, EfficientNet). An easy to compute flatness measure shows a clear correlation with test accuracy.